2024-03-29T10:32:02.357 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:32:02.357 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology --verbose --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649/orig.config.yaml 2024-03-29T10:32:02.391 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:32:02.477 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622649' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOs/ndKlvjGdBzG5f1iS4hR4q57LTrq2aAzoPkLTTFv0bqaek2BKZukHVQyRNNKa+it/lpQSm/z9NtWS+GyXZ3U= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwqt2fg/uG9aD+S9vmfcYKVAE4utlnhEcAyqIpN44VF5msL/7BqUAgmNf3en7xQFd8nKxjiBt19L8OdGwTYbNM= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:32:02.477 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:32:02.478 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:32:02.478 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:32:02.480 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:32:02.507 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:32:02.507 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:32:02.507 INFO:teuthology.packaging:ref: None 2024-03-29T10:32:02.507 INFO:teuthology.packaging:tag: None 2024-03-29T10:32:02.507 INFO:teuthology.packaging:branch: main 2024-03-29T10:32:02.507 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:32:02.507 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:32:02.942 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:32:02.943 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:32:02.950 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:32:02.950 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:32:03.021 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:32:03.033 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:32:03.039 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:32:03.062 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:26:54.716326', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOs/ndKlvjGdBzG5f1iS4hR4q57LTrq2aAzoPkLTTFv0bqaek2BKZukHVQyRNNKa+it/lpQSm/z9NtWS+GyXZ3U='} 2024-03-29T10:32:03.088 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:26:54.717795', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwqt2fg/uG9aD+S9vmfcYKVAE4utlnhEcAyqIpN44VF5msL/7BqUAgmNf3en7xQFd8nKxjiBt19L8OdGwTYbNM='} 2024-03-29T10:32:03.088 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:32:03.094 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:32:03.094 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:32:03.094 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:32:03.163 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1d3c5780d0>, signals=[15]) 2024-03-29T10:32:03.163 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:32:03.169 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:32:03.169 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T10:32:03.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:32:03.264 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-03-29T10:32:03.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:32:03.354 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:32:03.359 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-03-29T10:32:03.377 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-03-29T10:32:03.377 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-03-29T10:32:03.433 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-03-29T10:32:03.433 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-03-29T10:32:03.434 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-03-29T10:32:03.434 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:32:03.434 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-03-29T10:32:03.434 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:32:03.434 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:32:03.434 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-03-29T10:32:03.434 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:32:03.434 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:32:03.434 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:32:03.434 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:32:03.435 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:32:03.435 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:32:03.435 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-03-29T10:32:03.459 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-03-29T10:32:03.476 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-03-29T10:32:03.477 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-03-29T10:32:03.536 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-03-29T10:32:03.536 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-03-29T10:32:03.536 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-03-29T10:32:03.536 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:32:03.536 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-03-29T10:32:03.536 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:32:03.536 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:32:03.536 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-03-29T10:32:03.536 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:32:03.537 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:32:03.537 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:32:03.537 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:32:03.537 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:32:03.537 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:32:03.537 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-03-29T10:32:03.576 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:32:03.585 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:32:03.594 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:32:03.595 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:32:03.597 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:32:03.612 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:32:03.618 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:32:03.618 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:32:03.653 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:32:03.806 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:32:03.862 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:32:03.876 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:32:03.876 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:32:03.876 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:32:03.877 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:32:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:32:03.892 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-03-29T10:32:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:32:03.893 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-29T10:32:03.951 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:32:03.951 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-361.el9.x86_64 2024-03-29T10:32:03.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-03-29T10:32:03.953 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:32:03.953 INFO:teuthology.task.kernel:Running kernel on smithi189: 5.14.0-361.el9.x86_64 2024-03-29T10:32:03.953 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum install -y kernel 2024-03-29T10:32:06.184 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T10:32:06.235 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-29T10:32:07.610 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-29T10:32:07.643 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-29T10:32:10.943 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:03 2024-03-29T10:32:13.552 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-29T10:32:16.915 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 920 kB/s | 8.0 MB 00:08 2024-03-29T10:32:17.544 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-29T10:32:19.029 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-29T10:32:19.983 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 19 MB 00:01 2024-03-29T10:32:21.091 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T10:32:24.592 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 6.1 MB 00:01 2024-03-29T10:32:25.963 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-29T10:32:26.109 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-29T10:32:27.427 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T10:32:28.270 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:32:28.270 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:32:28.357 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:32:28.363 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:32:28.363 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T10:32:28.363 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:32:28.363 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:32:28.363 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:32:28.363 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T10:32:28.363 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:32:28.363 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:32:28.364 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:32:28.364 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:32:28.364 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:32:28.364 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:32:28.364 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-03-29T10:32:28.364 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:32:28.364 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 76 M 2024-03-29T10:32:28.365 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 123 M 2024-03-29T10:32:28.365 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:32:28.539 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T10:32:29.048 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 22 MB/s | 15 MB 00:00 2024-03-29T10:32:29.656 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 26 MB/s | 34 MB 00:01 2024-03-29T10:32:30.148 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T10:32:30.151 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:30.151 INFO:teuthology.orchestra.run.smithi098.stdout:Total 43 MB/s | 76 MB 00:01 2024-03-29T10:32:30.152 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:32:30.529 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:32:30.529 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:32:31.356 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:32:31.357 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:32:32.452 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T10:32:33.676 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:32:34.274 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:32:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:32:34.725 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:32:34.726 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:32:34.820 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:32:34.830 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:32:34.830 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-03-29T10:32:34.830 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:32:34.830 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:32:34.830 INFO:teuthology.orchestra.run.smithi189.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:32:34.830 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-29T10:32:34.830 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:32:34.830 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:32:34.830 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:32:34.830 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:32:34.830 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:32:34.830 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:32:34.831 INFO:teuthology.orchestra.run.smithi189.stdout:Install 4 Packages 2024-03-29T10:32:34.831 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:32:34.831 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 76 M 2024-03-29T10:32:34.831 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 123 M 2024-03-29T10:32:34.831 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:32:34.978 INFO:teuthology.orchestra.run.smithi189.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:32:35.779 INFO:teuthology.orchestra.run.smithi189.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 16 MB/s | 15 MB 00:00 2024-03-29T10:32:36.304 INFO:teuthology.orchestra.run.smithi189.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 23 MB/s | 34 MB 00:01 2024-03-29T10:32:36.604 INFO:teuthology.orchestra.run.smithi189.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:36.607 INFO:teuthology.orchestra.run.smithi189.stdout:Total 43 MB/s | 76 MB 00:01 2024-03-29T10:32:36.608 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:32:37.009 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:32:37.009 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:32:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:32:37.246 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:32:37.846 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:32:37.846 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:32:40.176 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:32:40.770 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:32:40.855 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:32:41.960 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:32:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:32:43.616 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:32:43.748 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:32:46.770 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:32:48.470 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:32:48.507 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:32:53.283 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:33:16.268 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:33:16.268 INFO:teuthology.orchestra.run.smithi098.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:33:16.268 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:33:16.311 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:33:16.712 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:33:16.712 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:33:16.712 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:33:16.712 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:33:17.150 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:33:17.150 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:33:17.150 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:33:17.150 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:33:17.150 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:33:17.150 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:33:17.150 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:33:17.150 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:33:17.150 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:33:17.496 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:33:17.597 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:33:17.597 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:33:17.597 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:33:17.598 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:33:17.598 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:33:17.598 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:33:17.598 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:33:17.598 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:33:17.677 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:33:17.678 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:33:17.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-03-29T10:33:17.711 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-03-29T10:33:17.711 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 68.el9 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8264254 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:33:17.712 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-03-29T10:33:17.713 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-03-29T10:33:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:33:17.713 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:33:17.713 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:33:17.713 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:33:17.713 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-03-29T10:33:17.713 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:33:17.713 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:33:17.714 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:33:17.714 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:33:17.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:33:18.336 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-03-29T10:33:19.738 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:33:19.767 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-03-29T10:33:19.769 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-29T10:33:19.785 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.SjqnT4lZD0 2024-03-29T10:33:19.786 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SjqnT4lZD0 2024-03-29T10:33:19.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.SjqnT4lZD0 2024-03-29T10:33:19.984 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.SjqnT4lZD0 is 6KB 2024-03-29T10:33:20.006 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.SjqnT4lZD0 2024-03-29T10:33:20.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:33:20.087 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:33:20.087 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:33:20.087 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:33:20.087 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:33:20.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:33:20.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-03-29T10:33:22.749 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:33:22.749 INFO:teuthology.orchestra.run.smithi189.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:33:22.749 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:33:22.793 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:33:23.167 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:33:23.167 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:33:23.167 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:33:23.168 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:33:23.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:33:23.591 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:33:23.591 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:33:23.591 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:33:23.591 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:33:23.591 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:33:23.591 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:33:23.592 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:33:23.592 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:33:23.947 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:33:24.060 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:33:24.060 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:33:24.060 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:33:24.061 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:33:24.061 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:33:24.061 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:33:24.061 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi189.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:33:24.061 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:33:24.141 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:33:24.142 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:33:24.142 DEBUG:teuthology.orchestra.run.smithi189:> sudo rpm -qi grub2-tools 2024-03-29T10:33:24.187 INFO:teuthology.orchestra.run.smithi189.stdout:Name : grub2-tools 2024-03-29T10:33:24.188 INFO:teuthology.orchestra.run.smithi189.stdout:Epoch : 1 2024-03-29T10:33:24.188 INFO:teuthology.orchestra.run.smithi189.stdout:Version : 2.06 2024-03-29T10:33:24.188 INFO:teuthology.orchestra.run.smithi189.stdout:Release : 68.el9 2024-03-29T10:33:24.188 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: x86_64 2024-03-29T10:33:24.188 INFO:teuthology.orchestra.run.smithi189.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:33:24.188 INFO:teuthology.orchestra.run.smithi189.stdout:Group : Unspecified 2024-03-29T10:33:24.188 INFO:teuthology.orchestra.run.smithi189.stdout:Size : 8264254 2024-03-29T10:33:24.188 INFO:teuthology.orchestra.run.smithi189.stdout:License : GPLv3+ 2024-03-29T10:33:24.188 INFO:teuthology.orchestra.run.smithi189.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:33:24.188 INFO:teuthology.orchestra.run.smithi189.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:33:24.188 INFO:teuthology.orchestra.run.smithi189.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:33:24.188 INFO:teuthology.orchestra.run.smithi189.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:33:24.189 INFO:teuthology.orchestra.run.smithi189.stdout:Packager : builder@centos.org 2024-03-29T10:33:24.189 INFO:teuthology.orchestra.run.smithi189.stdout:Vendor : CentOS 2024-03-29T10:33:24.189 INFO:teuthology.orchestra.run.smithi189.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:33:24.189 INFO:teuthology.orchestra.run.smithi189.stdout:Summary : Support tools for GRUB. 2024-03-29T10:33:24.189 INFO:teuthology.orchestra.run.smithi189.stdout:Description : 2024-03-29T10:33:24.189 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:33:24.189 INFO:teuthology.orchestra.run.smithi189.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:33:24.189 INFO:teuthology.orchestra.run.smithi189.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:33:24.189 INFO:teuthology.orchestra.run.smithi189.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:33:24.189 INFO:teuthology.orchestra.run.smithi189.stdout:hardware devices. 2024-03-29T10:33:24.189 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:33:24.189 INFO:teuthology.orchestra.run.smithi189.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:33:24.190 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:33:24.190 INFO:teuthology.task.kernel:Updating grub on smithi189 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:33:24.191 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:33:24.767 INFO:teuthology.orchestra.run.smithi189.stderr:Generating grub configuration file ... 2024-03-29T10:33:26.139 INFO:teuthology.orchestra.run.smithi189.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:33:26.171 INFO:teuthology.orchestra.run.smithi189.stderr:done 2024-03-29T10:33:26.173 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-03-29T10:33:26.188 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.cztDgBqRtU 2024-03-29T10:33:26.189 DEBUG:teuthology.orchestra.run.smithi189:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cztDgBqRtU 2024-03-29T10:33:26.257 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.cztDgBqRtU 2024-03-29T10:33:26.398 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.cztDgBqRtU is 6KB 2024-03-29T10:33:26.447 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.cztDgBqRtU 2024-03-29T10:33:26.461 DEBUG:teuthology.orchestra.run.smithi189:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:33:26.525 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:33:26.525 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:33:26.525 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:33:26.525 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:33:26.527 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:33:26.681 DEBUG:teuthology.orchestra.run.smithi189:> sudo shutdown -r now 2024-03-29T10:33:50.270 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:33:50.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-29T10:33:50.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:33:56.690 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:33:56.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-29T10:33:56.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:34:08.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-29T10:34:15.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-03-29T10:34:17.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-29T10:34:17.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:34:20.862 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-29T10:34:24.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-29T10:34:24.204 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:34:27.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-03-29T10:34:32.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-29T10:34:32.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:34:39.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-29T10:34:39.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:34:57.695 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-03-29T10:35:04.767 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-29T10:35:05.352 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-29T10:35:05.352 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:35:05.352 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-29T10:35:05.366 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:35:05.366 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:35:05.366 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:35:05.366 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:35:06.367 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:35:06.367 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:35:06.442 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-03-29T10:35:06.479 DEBUG:teuthology.parallel:result is None 2024-03-29T10:35:12.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-29T10:35:12.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:35:13.056 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-03-29T10:35:13.488 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-29T10:35:13.488 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:35:13.489 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-03-29T10:35:13.503 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:35:13.503 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:35:13.503 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:35:13.503 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:35:14.504 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:35:14.505 DEBUG:teuthology.orchestra.run.smithi189:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:35:14.565 INFO:teuthology.orchestra.run.smithi189.stdout:ttyS1 2024-03-29T10:35:14.606 DEBUG:teuthology.parallel:result is None 2024-03-29T10:35:14.606 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:35:14.613 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:35:14.614 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:35:14.616 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:35:14.632 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:35:14.638 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:35:14.648 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:35:14.648 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:35:14.674 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:35:14.796 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:35:14.802 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:35:14.803 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:35:14.805 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:35:14.836 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:35:14.842 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:35:14.849 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:35:14.856 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:35:14.858 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:35:14.866 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:35:14.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:35:14.900 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:35:14.928 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:35:14.937 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:35:14.937 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:35:14.971 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:35:14.985 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:35:15.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:35:15.114 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:35:15.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:35:15.240 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:35:15.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:35:15.308 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:35:15.351 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:35:15.402 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:35:15.485 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:35:15.526 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:35:15.526 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:35:15.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-03-29T10:35:15.595 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-03-29T10:35:15.650 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:35:15.685 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:35:16.155 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:35:16.162 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:35:16.162 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:35:16.172 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:35:16.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-03-29T10:35:16.236 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-03-29T10:35:16.237 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-03-29T10:35:16.313 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-03-29T10:35:16.315 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:35:16.315 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-03-29T10:35:16.349 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-03-29T10:35:16.349 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-03-29T10:35:16.369 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-03-29T10:35:16.369 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:35:16.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:35:16.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:35:16.418 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:35:16.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:35:16.446 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:35:16.446 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:35:16.493 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:35:16.538 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:35:16.547 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T10:35:16.552 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T10:35:16.682 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:35:16.698 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:35:16.700 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:39:29.254 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-03-29T10:39:29.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-29T10:39:29.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:39:29.348 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-29T10:39:29.426 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-29T10:39:29.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-29T10:39:29.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:39:29.518 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-03-29T10:39:29.602 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-29T10:39:29.602 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:39:29.614 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:39:29.614 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:39:29.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:39:29.617 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:39:29.617 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:39:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:39:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:39:29.674 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:39:29.691 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:39:29.694 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-03-29T10:39:29.707 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-03-29T10:39:29.720 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-03-29T10:39:29.723 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:39:29.734 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-03-29T10:39:29.740 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:39:29.751 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:39:29.767 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:39:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:39:29.795 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:39:29.795 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-29T10:39:29.795 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:39:29.795 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:39:29.795 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:39:29.795 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:39:29.830 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:39:29.834 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:39:29.834 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-03-29T10:39:29.834 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:39:29.834 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:39:29.834 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:39:29.834 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:39:29.834 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:39:29.843 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:39:29.843 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:39:29.843 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:39:29.843 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:39:29.850 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:39:29.850 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:39:29.851 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:39:29.851 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:39:29.853 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:39:29.854 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:39:30.004 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:39:30.004 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:39:30.080 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:39:30.080 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:39:30.195 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:39:30.196 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:39:30.196 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:39:30.224 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:39:30.224 DEBUG:teuthology.orchestra.run.smithi189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:39:30.292 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:39:30.292 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:39:30.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:39:30.294 DEBUG:teuthology.orchestra.run.smithi189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:39:30.322 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:39:30.322 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:39:30.374 DEBUG:teuthology.orchestra.run.smithi189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:39:30.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:39:30.441 INFO:teuthology.orchestra.run.smithi189.stdout:check_obsoletes = 1 2024-03-29T10:39:30.442 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum clean all 2024-03-29T10:39:30.470 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:39:30.536 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-03-29T10:39:30.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-03-29T10:39:30.681 INFO:teuthology.orchestra.run.smithi189.stdout:74 files removed 2024-03-29T10:39:30.708 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-radosgw 2024-03-29T10:39:30.788 INFO:teuthology.orchestra.run.smithi098.stdout:74 files removed 2024-03-29T10:39:30.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw 2024-03-29T10:39:31.644 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 156 kB/s | 82 kB 00:00 2024-03-29T10:39:31.755 INFO:teuthology.orchestra.run.smithi189.stdout:ceph packages for x86_64 110 kB/s | 82 kB 00:00 2024-03-29T10:39:32.078 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-29T10:39:32.189 INFO:teuthology.orchestra.run.smithi189.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-29T10:39:32.605 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-03-29T10:39:32.656 INFO:teuthology.orchestra.run.smithi189.stdout:ceph source packages 4.4 kB/s | 1.9 kB 00:00 2024-03-29T10:39:33.314 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 9.9 MB/s | 6.7 MB 00:00 2024-03-29T10:39:33.408 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - AppStream 9.4 MB/s | 6.7 MB 00:00 2024-03-29T10:39:35.098 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2024-03-29T10:39:35.209 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2024-03-29T10:39:36.297 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 8.0 MB 00:00 2024-03-29T10:39:38.557 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-29T10:39:44.341 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.1 MB 00:02 2024-03-29T10:39:44.640 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 904 kB/s | 8.0 MB 00:09 2024-03-29T10:39:45.750 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 16 kB 00:00 2024-03-29T10:39:47.114 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T10:39:52.077 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-29T10:39:54.714 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:39:54.736 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:39:54.736 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T10:39:54.736 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:39:54.736 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:39:54.736 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:39:54.736 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-03-29T10:39:54.736 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:39:54.737 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:39:54.737 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T10:39:54.737 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options 2024-03-29T10:39:54.737 INFO:teuthology.orchestra.run.smithi098.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:39:54.737 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:39:54.737 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:39:54.737 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:39:54.737 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:39:54.737 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:39:54.737 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:39:54.737 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:39:54.737 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:39:54.737 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:39:54.738 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:39:54.738 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:39:54.738 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:39:54.738 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:39:54.738 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:39:54.738 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:39:54.738 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:39:54.738 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse 2024-03-29T10:39:54.738 INFO:teuthology.orchestra.run.smithi098.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:39:54.738 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:39:54.738 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:39:54.738 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:39:54.738 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout:Install 30 Packages 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 2 Packages 2024-03-29T10:39:54.739 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:39:54.740 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 58 M 2024-03-29T10:39:54.740 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:39:55.711 INFO:teuthology.orchestra.run.smithi098.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 8.7 MB/s | 5.3 MB 00:00 2024-03-29T10:39:55.761 INFO:teuthology.orchestra.run.smithi098.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 506 kB/s | 25 kB 00:00 2024-03-29T10:39:55.878 INFO:teuthology.orchestra.run.smithi098.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 6.2 MB/s | 742 kB 00:00 2024-03-29T10:39:55.987 INFO:teuthology.orchestra.run.smithi098.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.5 MB/s | 498 kB 00:00 2024-03-29T10:39:56.579 INFO:teuthology.orchestra.run.smithi098.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 8.6 MB/s | 5.1 MB 00:00 2024-03-29T10:39:56.688 INFO:teuthology.orchestra.run.smithi098.stdout:(6/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 6.1 MB/s | 9.7 MB 00:01 2024-03-29T10:39:56.721 INFO:teuthology.orchestra.run.smithi098.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 318 kB/s | 45 kB 00:00 2024-03-29T10:39:56.772 INFO:teuthology.orchestra.run.smithi098.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.2 MB/s | 162 kB 00:00 2024-03-29T10:39:56.797 INFO:teuthology.orchestra.run.smithi098.stdout:(9/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.2 MB/s | 131 kB 00:00 2024-03-29T10:39:56.847 INFO:teuthology.orchestra.run.smithi098.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 4.2 MB/s | 320 kB 00:00 2024-03-29T10:39:56.889 INFO:teuthology.orchestra.run.smithi098.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 2.3 MB/s | 99 kB 00:00 2024-03-29T10:39:56.990 INFO:teuthology.orchestra.run.smithi098.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.5 MB/s | 298 kB 00:00 2024-03-29T10:39:57.115 INFO:teuthology.orchestra.run.smithi098.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 470 kB/s | 106 kB 00:00 2024-03-29T10:39:57.140 INFO:teuthology.orchestra.run.smithi098.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-29T10:39:57.165 INFO:teuthology.orchestra.run.smithi098.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-29T10:39:57.216 INFO:teuthology.orchestra.run.smithi098.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-29T10:39:57.241 INFO:teuthology.orchestra.run.smithi098.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T10:39:57.266 INFO:teuthology.orchestra.run.smithi098.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-03-29T10:39:57.308 INFO:teuthology.orchestra.run.smithi098.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T10:39:57.334 INFO:teuthology.orchestra.run.smithi098.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 523 kB/s | 35 kB 00:00 2024-03-29T10:39:57.367 INFO:teuthology.orchestra.run.smithi098.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 793 kB/s | 46 kB 00:00 2024-03-29T10:39:57.559 INFO:teuthology.orchestra.run.smithi098.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 8.1 MB/s | 20 MB 00:02 2024-03-29T10:39:58.001 INFO:teuthology.orchestra.run.smithi098.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 66 kB/s | 26 kB 00:00 2024-03-29T10:39:58.168 INFO:teuthology.orchestra.run.smithi098.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 370 kB/s | 309 kB 00:00 2024-03-29T10:39:58.194 INFO:teuthology.orchestra.run.smithi098.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-03-29T10:39:58.285 INFO:teuthology.orchestra.run.smithi098.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 575 kB/s | 67 kB 00:00 2024-03-29T10:39:58.427 INFO:teuthology.orchestra.run.smithi098.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:39:58.603 INFO:teuthology.orchestra.run.smithi098.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 4.4 MB 00:01 2024-03-29T10:39:58.636 INFO:teuthology.orchestra.run.smithi098.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.9 MB/s | 838 kB 00:00 2024-03-29T10:39:58.770 INFO:teuthology.orchestra.run.smithi098.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.6 MB/s | 1.6 MB 00:00 2024-03-29T10:39:59.045 INFO:teuthology.orchestra.run.smithi098.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 7.6 MB/s | 3.4 MB 00:00 2024-03-29T10:39:59.104 INFO:teuthology.orchestra.run.smithi098.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 6.7 MB/s | 3.1 MB 00:00 2024-03-29T10:39:59.110 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:39:59.110 INFO:teuthology.orchestra.run.smithi098.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-29T10:39:59.110 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:39:59.148 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:39:59.148 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:39:59.626 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:39:59.626 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:40:01.102 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:40:01.179 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:40:01.217 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:40:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:40:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:40:01.302 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:40:01.466 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:40:01.520 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:40:01.849 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:40:01.903 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:40:01.987 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:40:02.088 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:40:02.114 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:40:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:40:02.205 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:40:02.253 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:40:02.317 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:40:02.341 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:40:02.391 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:40:02.442 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:40:02.566 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:40:02.598 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:40:02.645 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:40:02.672 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:40:02.701 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:40:02.733 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:40:03.055 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:40:03.080 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:40:03.169 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:40:03.201 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:40:03.245 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:40:03.709 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:40:03.741 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:40:03.804 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:40:03.872 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:40:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:40:06.116 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:40:06.149 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:40:06.430 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 934 kB/s | 19 MB 00:20 2024-03-29T10:40:06.701 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:40:06.735 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:40:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:40:06.829 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:40:06.829 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:40:06.829 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:06.870 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:40:10.542 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.1 MB 00:00 2024-03-29T10:40:12.407 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-29T10:40:13.475 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-29T10:40:16.523 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:40:16.523 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-03-29T10:40:16.523 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-03-29T10:40:16.523 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-03-29T10:40:16.524 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-03-29T10:40:16.524 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-03-29T10:40:16.524 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-03-29T10:40:16.524 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-03-29T10:40:16.524 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:17.435 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:40:17.460 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:40:17.460 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:40:17.460 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:40:17.460 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:40:17.460 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:40:17.460 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:17.462 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:40:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:40:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:40:18.375 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-29T10:40:18.744 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:40:18.744 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:40:18.744 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:40:18.744 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:40:18.744 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:40:18.745 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:40:18.745 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:40:18.745 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:40:18.745 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:40:18.745 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:40:18.745 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:40:18.745 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:40:18.745 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:40:18.745 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:40:18.745 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:40:18.745 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:40:18.745 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:40:18.745 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:40:18.746 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:40:18.746 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:40:18.746 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:40:18.746 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:40:18.746 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:40:18.746 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:40:18.746 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:40:18.746 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:40:18.746 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:40:18.746 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:40:18.747 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:40:18.747 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:40:18.747 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:40:18.747 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:40:18.747 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:40:18.747 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:40:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:40:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:19.971 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-03-29T10:40:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.971 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:40:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:40:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:40:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:40:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:40:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:40:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:40:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:40:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:40:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:40:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:19.973 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:40:20.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-test 2024-03-29T10:40:20.895 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:40:20.992 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:40:21.004 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:40:21.004 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrading: 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:40:21.005 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:40:21.006 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout:Installing weak dependencies: 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:40:21.007 INFO:teuthology.orchestra.run.smithi189.stdout:Install 30 Packages 2024-03-29T10:40:21.008 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrade 2 Packages 2024-03-29T10:40:21.008 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:40:21.008 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 58 M 2024-03-29T10:40:21.008 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:40:21.047 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:40:21.047 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:40:21.047 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-03-29T10:40:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:21.049 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 48 M 2024-03-29T10:40:21.049 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 204 M 2024-03-29T10:40:21.049 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:40:21.273 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:40:21.323 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:40:21.357 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:40:21.416 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 537 kB/s | 163 kB 00:00 2024-03-29T10:40:21.516 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T10:40:21.726 INFO:teuthology.orchestra.run.smithi189.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 8.4 MB/s | 5.3 MB 00:00 2024-03-29T10:40:21.776 INFO:teuthology.orchestra.run.smithi189.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 504 kB/s | 25 kB 00:00 2024-03-29T10:40:21.893 INFO:teuthology.orchestra.run.smithi189.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 6.2 MB/s | 742 kB 00:00 2024-03-29T10:40:21.985 INFO:teuthology.orchestra.run.smithi189.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.3 MB/s | 498 kB 00:00 2024-03-29T10:40:22.527 INFO:teuthology.orchestra.run.smithi189.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 9.4 MB/s | 5.1 MB 00:00 2024-03-29T10:40:22.594 INFO:teuthology.orchestra.run.smithi189.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 676 kB/s | 45 kB 00:00 2024-03-29T10:40:22.661 INFO:teuthology.orchestra.run.smithi189.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.9 MB/s | 131 kB 00:00 2024-03-29T10:40:22.736 INFO:teuthology.orchestra.run.smithi189.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.1 MB/s | 162 kB 00:00 2024-03-29T10:40:22.820 INFO:teuthology.orchestra.run.smithi189.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.8 MB/s | 320 kB 00:00 2024-03-29T10:40:22.904 INFO:teuthology.orchestra.run.smithi189.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.5 MB/s | 298 kB 00:00 2024-03-29T10:40:22.979 INFO:teuthology.orchestra.run.smithi189.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 99 kB 00:00 2024-03-29T10:40:23.146 INFO:teuthology.orchestra.run.smithi189.stdout:(12/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 4.7 MB/s | 9.7 MB 00:02 2024-03-29T10:40:23.180 INFO:teuthology.orchestra.run.smithi189.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 529 kB/s | 106 kB 00:00 2024-03-29T10:40:23.222 INFO:teuthology.orchestra.run.smithi189.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-29T10:40:23.247 INFO:teuthology.orchestra.run.smithi189.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 662 kB/s | 44 kB 00:00 2024-03-29T10:40:23.347 INFO:teuthology.orchestra.run.smithi189.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 661 kB 00:00 2024-03-29T10:40:23.372 INFO:teuthology.orchestra.run.smithi189.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2024-03-29T10:40:23.398 INFO:teuthology.orchestra.run.smithi189.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2024-03-29T10:40:23.423 INFO:teuthology.orchestra.run.smithi189.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T10:40:23.448 INFO:teuthology.orchestra.run.smithi189.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-03-29T10:40:23.473 INFO:teuthology.orchestra.run.smithi189.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 922 kB/s | 46 kB 00:00 2024-03-29T10:40:24.082 INFO:teuthology.orchestra.run.smithi189.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 488 kB/s | 309 kB 00:00 2024-03-29T10:40:24.182 INFO:teuthology.orchestra.run.smithi189.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-29T10:40:24.283 INFO:teuthology.orchestra.run.smithi189.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 488 kB/s | 49 kB 00:00 2024-03-29T10:40:24.383 INFO:teuthology.orchestra.run.smithi189.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 682 kB/s | 67 kB 00:00 2024-03-29T10:40:24.441 INFO:teuthology.orchestra.run.smithi189.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-03-29T10:40:24.516 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 14 MB/s | 47 MB 00:03 2024-03-29T10:40:24.519 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:40:24.520 INFO:teuthology.orchestra.run.smithi098.stdout:Total 14 MB/s | 48 MB 00:03 2024-03-29T10:40:24.535 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:40:24.556 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:40:24.556 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:40:24.608 INFO:teuthology.orchestra.run.smithi189.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.7 MB/s | 838 kB 00:00 2024-03-29T10:40:24.634 INFO:teuthology.orchestra.run.smithi189.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-03-29T10:40:24.742 INFO:teuthology.orchestra.run.smithi189.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-29T10:40:24.938 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:40:24.938 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:40:24.968 INFO:teuthology.orchestra.run.smithi189.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 10 MB/s | 3.4 MB 00:00 2024-03-29T10:40:25.218 INFO:teuthology.orchestra.run.smithi189.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 6.6 MB/s | 3.1 MB 00:00 2024-03-29T10:40:25.525 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:40:25.591 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:40:25.634 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:40:25.675 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:40:25.725 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:40:25.753 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:40:26.885 INFO:teuthology.orchestra.run.smithi189.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 3.4 MB/s | 20 MB 00:05 2024-03-29T10:40:26.890 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:40:26.890 INFO:teuthology.orchestra.run.smithi189.stdout:Total 9.8 MB/s | 58 MB 00:05 2024-03-29T10:40:26.891 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:40:26.930 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:40:26.930 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:40:27.412 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:40:27.412 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:40:28.874 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:40:28.961 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:40:28.998 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:40:29.023 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:40:29.046 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:40:29.063 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:40:29.256 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:40:29.319 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:40:29.647 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:40:29.708 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:40:29.782 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:40:29.871 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:40:29.893 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:40:29.956 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:40:29.984 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:40:30.020 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:40:30.087 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:40:30.112 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:40:30.176 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:40:30.211 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:40:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:40:30.302 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:40:30.318 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:40:30.353 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:40:30.405 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:40:30.434 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:40:30.463 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:40:30.496 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:40:30.813 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:40:30.843 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:40:30.925 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:40:30.957 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:40:31.002 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:40:31.464 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:40:31.488 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:40:31.558 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:40:31.627 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:40:31.656 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:40:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:40:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:40:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:40:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:40:32.660 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:40:32.661 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:40:33.037 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:40:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:33.038 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:40:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:40:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:40:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:40:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:40:33.038 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:33.038 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:40:33.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph 2024-03-29T10:40:33.816 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:40:33.853 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:40:33.933 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:41 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:40:34.102 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:40:34.104 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================= 2024-03-29T10:40:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T10:40:34.105 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================= 2024-03-29T10:40:34.105 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:40:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:40:34.105 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T10:40:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:40:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:40:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:40:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:40:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:40:34.105 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:40:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:40:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:40:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:40:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:40:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:40:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:40:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:40:34.106 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:40:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:40:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:40:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:40:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:40:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:40:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:40:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:40:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:40:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:40:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:40:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:40:34.107 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:40:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:40:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:40:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:40:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:40:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:40:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:40:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:40:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:40:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:40:34.108 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:34.108 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:40:34.109 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================= 2024-03-29T10:40:34.109 INFO:teuthology.orchestra.run.smithi098.stdout:Install 36 Packages 2024-03-29T10:40:34.109 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:34.110 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 30 M 2024-03-29T10:40:34.110 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 107 M 2024-03-29T10:40:34.110 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:40:34.400 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:40:34.439 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:40:34.538 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:40:34.538 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:40:34.539 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:40:34.539 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:40:34.574 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:40:34.654 INFO:teuthology.orchestra.run.smithi098.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 34 kB/s | 6.6 kB 00:00 2024-03-29T10:40:34.854 INFO:teuthology.orchestra.run.smithi098.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.4 MB/s | 2.1 MB 00:00 2024-03-29T10:40:35.179 INFO:teuthology.orchestra.run.smithi098.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 8.9 MB/s | 4.7 MB 00:00 2024-03-29T10:40:35.213 INFO:teuthology.orchestra.run.smithi098.stdout:(4/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 1.5 MB 00:00 2024-03-29T10:40:35.238 INFO:teuthology.orchestra.run.smithi098.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 4.1 MB/s | 247 kB 00:00 2024-03-29T10:40:35.356 INFO:teuthology.orchestra.run.smithi098.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T10:40:35.381 INFO:teuthology.orchestra.run.smithi098.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2024-03-29T10:40:35.406 INFO:teuthology.orchestra.run.smithi098.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 634 kB/s | 32 kB 00:00 2024-03-29T10:40:35.431 INFO:teuthology.orchestra.run.smithi098.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 939 kB/s | 47 kB 00:00 2024-03-29T10:40:35.456 INFO:teuthology.orchestra.run.smithi098.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-03-29T10:40:35.490 INFO:teuthology.orchestra.run.smithi098.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-29T10:40:35.549 INFO:teuthology.orchestra.run.smithi098.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-03-29T10:40:35.574 INFO:teuthology.orchestra.run.smithi098.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-29T10:40:35.599 INFO:teuthology.orchestra.run.smithi098.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T10:40:35.624 INFO:teuthology.orchestra.run.smithi098.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T10:40:35.850 INFO:teuthology.orchestra.run.smithi098.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 17 MB/s | 17 MB 00:00 2024-03-29T10:40:35.875 INFO:teuthology.orchestra.run.smithi098.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 89 kB/s | 22 kB 00:00 2024-03-29T10:40:35.917 INFO:teuthology.orchestra.run.smithi098.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 394 kB/s | 125 kB 00:00 2024-03-29T10:40:36.150 INFO:teuthology.orchestra.run.smithi098.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 158 kB/s | 43 kB 00:00 2024-03-29T10:40:36.217 INFO:teuthology.orchestra.run.smithi098.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 411 kB/s | 151 kB 00:00 2024-03-29T10:40:36.268 INFO:teuthology.orchestra.run.smithi098.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 490 kB/s | 172 kB 00:00 2024-03-29T10:40:36.310 INFO:teuthology.orchestra.run.smithi098.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-03-29T10:40:36.335 INFO:teuthology.orchestra.run.smithi098.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2024-03-29T10:40:36.360 INFO:teuthology.orchestra.run.smithi098.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-29T10:40:36.386 INFO:teuthology.orchestra.run.smithi098.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-29T10:40:36.411 INFO:teuthology.orchestra.run.smithi098.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-03-29T10:40:36.436 INFO:teuthology.orchestra.run.smithi098.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-03-29T10:40:36.462 INFO:teuthology.orchestra.run.smithi098.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 612 kB/s | 46 kB 00:00 2024-03-29T10:40:36.487 INFO:teuthology.orchestra.run.smithi098.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-29T10:40:36.521 INFO:teuthology.orchestra.run.smithi098.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-29T10:40:36.546 INFO:teuthology.orchestra.run.smithi098.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-29T10:40:36.580 INFO:teuthology.orchestra.run.smithi098.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 977 kB/s | 90 kB 00:00 2024-03-29T10:40:36.604 INFO:teuthology.orchestra.run.smithi098.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 427 kB/s | 36 kB 00:00 2024-03-29T10:40:36.630 INFO:teuthology.orchestra.run.smithi098.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-29T10:40:36.672 INFO:teuthology.orchestra.run.smithi098.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-29T10:40:36.697 INFO:teuthology.orchestra.run.smithi098.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-29T10:40:36.703 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:40:36.703 INFO:teuthology.orchestra.run.smithi098.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-29T10:40:36.725 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:40:36.764 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:40:36.764 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:40:37.161 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:40:37.162 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:40:38.483 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:40:38.533 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:40:38.568 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:40:38.602 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:40:38.672 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:40:38.709 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:40:38.742 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:40:38.834 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:40:38.919 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:40:38.961 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:40:39.041 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:40:39.095 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:40:39.134 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:40:39.191 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:40:39.246 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:40:39.438 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:40:39.497 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:40:39.527 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:40:39.562 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:40:39.592 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:40:39.670 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:40:39.710 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:40:39.752 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:40:39.792 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:40:39.828 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:40:39.862 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:40:39.942 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:40:39.985 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:40:40.056 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:40:40.192 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:40:40.310 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:40:40.470 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:40:40.507 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:40:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:40:40.533 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:40:40.533 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:40:40.533 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:40:40.533 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:40:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:41.991 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:40:42.017 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:40:42.017 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:40:42.017 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:40:42.017 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:40:42.017 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:40:42.018 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:42.467 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:40:42.490 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:40:42.491 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:40:42.491 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:40:42.491 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:40:42.491 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:40:42.491 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:42.701 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:40:42.723 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:40:42.724 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:40:42.724 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:40:42.724 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:40:42.724 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:40:42.724 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:42.790 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:40:44.182 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:40:44.182 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /sys 2024-03-29T10:40:44.182 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /proc 2024-03-29T10:40:44.182 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /mnt 2024-03-29T10:40:44.182 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /var/tmp 2024-03-29T10:40:44.182 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /home 2024-03-29T10:40:44.182 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /root 2024-03-29T10:40:44.182 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /tmp 2024-03-29T10:40:44.182 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:40:44.665 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:40:44.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:40:44.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:40:44.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:40:44.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:40:44.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:40:44.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:40:44.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:40:44.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:40:44.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:40:44.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:40:44.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:40:44.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:40:44.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:40:44.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:40:44.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:40:44.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:40:44.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:40:44.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:40:44.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:40:44.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:40:44.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:40:44.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:40:45.086 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:40:45.110 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:40:45.111 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:40:45.111 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:40:45.111 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:40:45.111 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:40:45.111 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:40:45.112 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:40:45.198 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:40:45.199 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:40:45.809 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:40:45.809 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:45.809 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:40:45.809 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:45.809 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:40:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:40:45.810 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:40:45.811 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:40:45.811 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:40:45.811 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:40:45.811 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:40:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:40:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:40:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:40:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:40:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:40:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:40:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:40:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:40:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:40:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:40:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:40:45.812 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:40:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:40:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:40:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:40:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:40:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:40:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:40:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:40:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:40:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:40:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:40:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:40:45.813 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:40:45.814 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:45.814 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:40:46.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base 2024-03-29T10:40:46.292 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:40:46.292 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:40:46.292 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:40:46.292 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:40:46.292 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:40:46.292 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:40:46.292 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:40:46.292 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:40:46.292 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:40:46.293 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:40:46.294 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:40:46.294 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:40:46.294 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:40:46.294 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:40:46.294 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:40:46.294 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:40:46.294 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:40:46.294 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:40:46.294 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:40:46.294 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:40:46.294 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:40:46.760 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:40:46.859 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:40:46.911 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:40:46.912 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T10:40:46.912 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:40:46.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install cephadm 2024-03-29T10:40:47.472 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:40:47.472 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:40:47.472 INFO:teuthology.orchestra.run.smithi189.stdout:Upgraded: 2024-03-29T10:40:47.472 INFO:teuthology.orchestra.run.smithi189.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.472 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.472 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:40:47.472 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:40:47.473 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.473 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.473 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.473 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.473 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:40:47.473 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:40:47.473 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:40:47.473 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.473 INFO:teuthology.orchestra.run.smithi189.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:40:47.473 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:40:47.473 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:40:47.473 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.474 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:40:47.474 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.474 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:40:47.474 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:40:47.474 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:40:47.474 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.474 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.474 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.474 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:40:47.474 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.474 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.474 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:47.475 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:40:47.475 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:40:47.475 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:40:47.475 INFO:teuthology.orchestra.run.smithi189.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:40:47.475 INFO:teuthology.orchestra.run.smithi189.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:40:47.475 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:40:47.475 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:40:47.543 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:40:47.694 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:40:47.695 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:40:47.695 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T10:40:47.695 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:40:47.695 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:40:47.695 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:40:47.695 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:47.696 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:40:47.696 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:40:47.696 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T10:40:47.696 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:47.696 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 747 k 2024-03-29T10:40:47.696 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 754 k 2024-03-29T10:40:47.696 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:40:47.740 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-test 2024-03-29T10:40:47.992 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.5 MB/s | 747 kB 00:00 2024-03-29T10:40:47.992 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:40:47.992 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.5 MB/s | 747 kB 00:00 2024-03-29T10:40:47.993 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:40:47.998 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:40:47.998 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:40:48.004 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:40:48.004 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:40:48.073 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:40:48.296 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:30 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:40:48.451 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:40:48.452 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:40:48.452 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-03-29T10:40:48.452 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:40:48.452 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:40:48.452 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:40:48.452 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-29T10:40:48.453 INFO:teuthology.orchestra.run.smithi189.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:40:48.453 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:40:48.453 INFO:teuthology.orchestra.run.smithi189.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:40:48.453 INFO:teuthology.orchestra.run.smithi189.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:40:48.453 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:40:48.453 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:40:48.453 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:40:48.453 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:40:48.453 INFO:teuthology.orchestra.run.smithi189.stdout:Install 6 Packages 2024-03-29T10:40:48.453 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:40:48.454 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 48 M 2024-03-29T10:40:48.454 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 204 M 2024-03-29T10:40:48.454 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:40:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:40:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:40:48.693 INFO:teuthology.orchestra.run.smithi189.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T10:40:48.760 INFO:teuthology.orchestra.run.smithi189.stdout:(2/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 659 kB/s | 163 kB 00:00 2024-03-29T10:40:48.793 INFO:teuthology.orchestra.run.smithi189.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.9 MB/s | 300 kB 00:00 2024-03-29T10:40:48.818 INFO:teuthology.orchestra.run.smithi189.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-03-29T10:40:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:40:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:40:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:40:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:40:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:49.206 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:40:49.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:40:49.702 INFO:teuthology.orchestra.run.smithi189.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 206 kB/s | 187 kB 00:00 2024-03-29T10:40:49.972 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:40:50.125 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:40:50.126 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:40:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-29T10:40:50.126 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:40:50.126 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:40:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:40:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:50.127 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:40:50.127 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:40:50.127 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T10:40:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:50.127 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 145 k 2024-03-29T10:40:50.127 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 436 k 2024-03-29T10:40:50.127 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:40:50.528 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 363 kB/s | 145 kB 00:00 2024-03-29T10:40:50.528 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:40:50.528 INFO:teuthology.orchestra.run.smithi098.stdout:Total 362 kB/s | 145 kB 00:00 2024-03-29T10:40:50.528 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:40:50.536 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:40:50.536 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:40:50.588 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:40:50.588 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:40:50.729 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:40:50.795 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:40:50.820 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:40:50.820 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:40:50.820 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:40:50.820 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:51.052 INFO:teuthology.orchestra.run.smithi189.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 19 MB/s | 47 MB 00:02 2024-03-29T10:40:51.055 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:40:51.055 INFO:teuthology.orchestra.run.smithi189.stdout:Total 19 MB/s | 48 MB 00:02 2024-03-29T10:40:51.070 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:40:51.092 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:40:51.092 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:40:51.471 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:40:51.472 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:40:51.770 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:40:51.770 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:51.770 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:40:51.770 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:51.771 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:51.771 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:40:52.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr 2024-03-29T10:40:52.054 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:40:52.127 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:40:52.174 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:40:52.206 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:40:52.270 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:40:52.299 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:40:52.577 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:40:52.675 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:40:52.727 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:40:52.728 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T10:40:52.728 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:40:52.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:40:53.354 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:40:53.503 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:40:53.504 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-03-29T10:40:53.504 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T10:40:53.504 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-03-29T10:40:53.504 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:40:53.504 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:40:53.504 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T10:40:53.504 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:40:53.504 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:40:53.504 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:40:53.505 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:40:53.505 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:40:53.505 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:40:53.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:40:53.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:40:53.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:40:53.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:40:53.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:40:53.505 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:53.505 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:40:53.505 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-03-29T10:40:53.505 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-03-29T10:40:53.505 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:53.506 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 6.9 M 2024-03-29T10:40:53.506 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-03-29T10:40:53.506 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:40:54.032 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 79 kB/s | 15 kB 00:00 2024-03-29T10:40:54.091 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 125 kB/s | 31 kB 00:00 2024-03-29T10:40:54.199 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-29T10:40:54.224 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 238 kB 00:00 2024-03-29T10:40:54.375 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.3 MB/s | 1.8 MB 00:00 2024-03-29T10:40:54.592 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-03-29T10:40:54.617 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-29T10:40:54.751 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 13 MB/s | 2.0 MB 00:00 2024-03-29T10:40:54.776 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 910 kB/s | 144 kB 00:00 2024-03-29T10:40:54.810 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 529 kB/s | 31 kB 00:00 2024-03-29T10:40:54.843 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-29T10:40:55.960 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 490 kB/s | 863 kB 00:01 2024-03-29T10:40:55.965 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:40:55.965 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.8 MB/s | 6.9 MB 00:02 2024-03-29T10:40:55.986 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:40:56.005 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:40:56.005 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:40:56.182 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:40:56.182 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:40:56.696 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:40:56.750 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:40:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:40:56.821 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:40:56.822 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:40:56.867 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:40:56.898 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:40:56.997 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:40:57.122 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:40:57.208 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:40:57.261 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:40:57.290 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:40:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:40:57.564 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:40:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:40:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:40:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:40:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:40:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:40:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:40:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:40:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:40:58.458 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:40:58.459 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:40:58.459 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:40:58.459 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:40:58.459 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:40:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:40:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:58.852 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:40:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:40:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:40:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:40:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:40:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:40:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:40:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:40:58.852 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:40:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:40:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:40:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:40:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:40:58.853 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:58.853 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:40:59.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:40:59.189 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:40:59.189 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:40:59.189 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:40:59.189 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:40:59.189 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:40:59.189 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:40:59.563 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:40:59.564 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:40:59.564 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:40:59.564 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:59.564 INFO:teuthology.orchestra.run.smithi189.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:40:59.564 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:40:59.564 INFO:teuthology.orchestra.run.smithi189.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:40:59.564 INFO:teuthology.orchestra.run.smithi189.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:40:59.564 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:40:59.564 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:40:59.564 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:40:59.650 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:07 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:40:59.803 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:40:59.804 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-29T10:40:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T10:40:59.804 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-29T10:40:59.804 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:40:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:40:59.804 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T10:40:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:40:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:40:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:40:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:40:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:40:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:40:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:40:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:40:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:40:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:40:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:40:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:59.806 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:40:59.806 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-29T10:40:59.806 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-03-29T10:40:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:40:59.806 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 37 M 2024-03-29T10:40:59.806 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 209 M 2024-03-29T10:40:59.807 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:40:59.883 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph 2024-03-29T10:41:00.251 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 237 kB/s | 32 kB 00:00 2024-03-29T10:41:00.284 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 538 kB/s | 17 kB 00:00 2024-03-29T10:41:00.318 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T10:41:00.377 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-29T10:41:00.448 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:42 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:00.560 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T10:41:00.602 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:00.605 INFO:teuthology.orchestra.run.smithi189.stdout:========================================================================================= 2024-03-29T10:41:00.605 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-03-29T10:41:00.605 INFO:teuthology.orchestra.run.smithi189.stdout:========================================================================================= 2024-03-29T10:41:00.605 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:41:00.605 INFO:teuthology.orchestra.run.smithi189.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:41:00.605 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-29T10:41:00.605 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:41:00.605 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:41:00.605 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:41:00.605 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:41:00.606 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:41:00.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:41:00.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:41:00.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:41:00.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:41:00.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:41:00.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:41:00.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:41:00.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:41:00.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:41:00.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:41:00.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:41:00.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:41:00.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:41:00.608 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:41:00.608 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:41:00.608 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:41:00.608 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:41:00.608 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:00.608 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:41:00.608 INFO:teuthology.orchestra.run.smithi189.stdout:========================================================================================= 2024-03-29T10:41:00.608 INFO:teuthology.orchestra.run.smithi189.stdout:Install 36 Packages 2024-03-29T10:41:00.608 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:00.609 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 30 M 2024-03-29T10:41:00.609 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 107 M 2024-03-29T10:41:00.609 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:41:00.627 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T10:41:00.653 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T10:41:00.953 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-29T10:41:01.340 INFO:teuthology.orchestra.run.smithi189.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-03-29T10:41:01.445 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 397 kB/s | 195 kB 00:00 2024-03-29T10:41:01.482 INFO:teuthology.orchestra.run.smithi189.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-03-29T10:41:01.587 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 862 kB/s | 803 kB 00:00 2024-03-29T10:41:01.649 INFO:teuthology.orchestra.run.smithi189.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.8 MB/s | 1.5 MB 00:00 2024-03-29T10:41:01.670 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-29T10:41:01.741 INFO:teuthology.orchestra.run.smithi189.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.6 MB/s | 247 kB 00:00 2024-03-29T10:41:01.858 INFO:teuthology.orchestra.run.smithi189.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T10:41:01.900 INFO:teuthology.orchestra.run.smithi189.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:41:01.933 INFO:teuthology.orchestra.run.smithi189.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2024-03-29T10:41:01.967 INFO:teuthology.orchestra.run.smithi189.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:41:02.000 INFO:teuthology.orchestra.run.smithi189.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T10:41:02.043 INFO:teuthology.orchestra.run.smithi189.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T10:41:02.118 INFO:teuthology.orchestra.run.smithi189.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T10:41:02.129 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.7 MB/s | 7.4 MB 00:02 2024-03-29T10:41:02.131 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:02.132 INFO:teuthology.orchestra.run.smithi098.stdout:Total 16 MB/s | 37 MB 00:02 2024-03-29T10:41:02.151 INFO:teuthology.orchestra.run.smithi189.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T10:41:02.176 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:41:02.185 INFO:teuthology.orchestra.run.smithi189.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T10:41:02.201 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:41:02.201 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:41:02.218 INFO:teuthology.orchestra.run.smithi189.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T10:41:02.377 INFO:teuthology.orchestra.run.smithi189.stdout:(15/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 19 MB/s | 17 MB 00:00 2024-03-29T10:41:02.469 INFO:teuthology.orchestra.run.smithi189.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 244 kB/s | 22 kB 00:00 2024-03-29T10:41:02.566 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:41:02.566 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:41:02.703 INFO:teuthology.orchestra.run.smithi189.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 724 kB/s | 151 kB 00:00 2024-03-29T10:41:02.728 INFO:teuthology.orchestra.run.smithi189.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 245 kB/s | 125 kB 00:00 2024-03-29T10:41:02.762 INFO:teuthology.orchestra.run.smithi189.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 741 kB/s | 43 kB 00:00 2024-03-29T10:41:02.878 INFO:teuthology.orchestra.run.smithi189.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-03-29T10:41:02.929 INFO:teuthology.orchestra.run.smithi189.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-03-29T10:41:02.962 INFO:teuthology.orchestra.run.smithi189.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 733 kB/s | 172 kB 00:00 2024-03-29T10:41:02.979 INFO:teuthology.orchestra.run.smithi189.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-03-29T10:41:03.022 INFO:teuthology.orchestra.run.smithi189.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2024-03-29T10:41:03.047 INFO:teuthology.orchestra.run.smithi189.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 290 kB/s | 19 kB 00:00 2024-03-29T10:41:03.113 INFO:teuthology.orchestra.run.smithi189.stdout:(26/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.6 MB/s | 4.7 MB 00:01 2024-03-29T10:41:03.139 INFO:teuthology.orchestra.run.smithi189.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 168 kB/s | 20 kB 00:00 2024-03-29T10:41:03.164 INFO:teuthology.orchestra.run.smithi189.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 396 kB/s | 46 kB 00:00 2024-03-29T10:41:03.206 INFO:teuthology.orchestra.run.smithi189.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-29T10:41:03.223 INFO:teuthology.orchestra.run.smithi189.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-03-29T10:41:03.262 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:41:03.265 INFO:teuthology.orchestra.run.smithi189.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-29T10:41:03.290 INFO:teuthology.orchestra.run.smithi189.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 533 kB/s | 36 kB 00:00 2024-03-29T10:41:03.315 INFO:teuthology.orchestra.run.smithi189.stdout:(33/36): python3-more-itertools-8.12.0-2.el9.no 391 kB/s | 79 kB 00:00 2024-03-29T10:41:03.326 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:41:03.340 INFO:teuthology.orchestra.run.smithi189.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T10:41:03.358 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:41:03.374 INFO:teuthology.orchestra.run.smithi189.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-03-29T10:41:03.416 INFO:teuthology.orchestra.run.smithi189.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-03-29T10:41:03.419 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:03.419 INFO:teuthology.orchestra.run.smithi189.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-29T10:41:03.439 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:41:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:41:03.478 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:41:03.478 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:41:03.495 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:41:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:41:03.702 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:41:03.778 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:41:03.890 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:41:03.890 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:41:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:41:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:41:05.196 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:41:05.260 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:41:05.296 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:41:05.329 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:41:05.400 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:41:05.436 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:41:05.469 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:41:05.562 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:41:05.631 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:41:05.673 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:41:05.743 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:41:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:41:05.797 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:41:05.836 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:41:05.885 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:41:05.939 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:41:06.125 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:41:06.173 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:41:06.203 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:41:06.238 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:41:06.267 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:41:06.344 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:41:06.378 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:41:06.411 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:41:06.452 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:41:06.487 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:41:06.522 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:41:06.611 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:41:06.652 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:41:06.725 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:41:06.781 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:41:06.802 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:41:06.862 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:41:06.979 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:41:07.144 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:41:07.173 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:41:07.199 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:41:07.199 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:41:07.199 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:41:07.199 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:41:07.199 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:41:07.199 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:08.632 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:41:08.659 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:41:08.659 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:41:08.659 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:41:08.659 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:41:08.660 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:41:08.660 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:09.100 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:41:09.123 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:41:09.123 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:41:09.123 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:41:09.123 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:41:09.123 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:41:09.124 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:09.334 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:41:09.357 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:41:09.357 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:41:09.357 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:41:09.357 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:41:09.357 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:41:09.357 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:41:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:41:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:41:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:41:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:41:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:41:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:41:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:41:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:41:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:41:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:41:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:41:09.422 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:41:09.889 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:41:09.889 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:41:09.889 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:41:09.889 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:41:09.889 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:41:09.889 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:41:09.889 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:41:09.889 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:41:09.889 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:41:09.890 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:41:09.890 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:41:09.890 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:41:09.890 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:41:09.890 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:41:09.890 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:41:09.890 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:41:09.890 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:41:10.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:41:10.790 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:18 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:41:10.942 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:41:10.943 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-29T10:41:10.943 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T10:41:10.943 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-29T10:41:10.943 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:41:10.943 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:41:10.944 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T10:41:10.944 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:41:10.944 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:41:10.944 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:41:10.944 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:41:10.944 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:41:10.944 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:41:10.944 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:41:10.945 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:41:10.945 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:41:10.945 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:41:10.945 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:41:10.945 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:41:10.945 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:41:10.945 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:41:10.945 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-29T10:41:10.945 INFO:teuthology.orchestra.run.smithi098.stdout:Install 13 Packages 2024-03-29T10:41:10.945 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:41:10.946 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.0 M 2024-03-29T10:41:10.946 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-03-29T10:41:10.946 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:41:11.232 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:41:11.232 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:41:11.232 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:41:11.232 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:41:11.232 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:41:11.232 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:41:11.232 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:41:11.232 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:41:11.233 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:41:11.234 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:41:11.234 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:41:11.234 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:41:11.234 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:41:11.235 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:41:11.235 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:41:11.235 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:41:11.235 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:41:11.235 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:41:11.235 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:41:11.235 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:41:11.235 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:41:11.235 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:41:11.422 INFO:teuthology.orchestra.run.smithi098.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 148 kB/s | 18 kB 00:00 2024-03-29T10:41:11.455 INFO:teuthology.orchestra.run.smithi098.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 156 kB/s | 24 kB 00:00 2024-03-29T10:41:11.506 INFO:teuthology.orchestra.run.smithi098.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-29T10:41:11.531 INFO:teuthology.orchestra.run.smithi098.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:41:11.556 INFO:teuthology.orchestra.run.smithi098.stdout:(5/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 203 kB/s | 52 kB 00:00 2024-03-29T10:41:11.581 INFO:teuthology.orchestra.run.smithi098.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-29T10:41:11.607 INFO:teuthology.orchestra.run.smithi098.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 567 kB/s | 43 kB 00:00 2024-03-29T10:41:12.059 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:41:12.059 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:12.059 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:41:12.060 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:41:12.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:41:12.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:41:12.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:41:12.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:41:12.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:41:12.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:41:12.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:41:12.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:41:12.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:41:12.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:41:12.061 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:41:12.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:41:12.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:41:12.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:41:12.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:41:12.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:41:12.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:41:12.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:41:12.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:41:12.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:41:12.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:41:12.062 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:41:12.062 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:12.377 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-base 2024-03-29T10:41:12.962 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:13.060 INFO:teuthology.orchestra.run.smithi189.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:41:13.113 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:13.113 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-03-29T10:41:13.113 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:13.175 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install cephadm 2024-03-29T10:41:13.738 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:13.888 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:13.889 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:13.889 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-03-29T10:41:13.889 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:13.890 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:41:13.890 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:41:13.890 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:13.890 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:41:13.890 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:13.890 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-03-29T10:41:13.890 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:13.890 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 747 k 2024-03-29T10:41:13.890 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 754 k 2024-03-29T10:41:13.890 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:41:14.164 INFO:teuthology.orchestra.run.smithi189.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.7 MB/s | 747 kB 00:00 2024-03-29T10:41:14.165 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:14.165 INFO:teuthology.orchestra.run.smithi189.stdout:Total 2.7 MB/s | 747 kB 00:00 2024-03-29T10:41:14.165 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:41:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:41:14.170 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:41:14.175 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:41:14.176 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:41:14.251 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:41:14.696 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:41:14.776 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:41:15.169 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:41:15.446 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:41:15.446 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:15.446 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:41:15.446 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:41:15.446 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:15.447 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:15.644 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:41:16.215 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:16.369 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:16.371 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:16.371 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repo Size 2024-03-29T10:41:16.371 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:16.371 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:41:16.371 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:41:16.372 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:16.372 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:41:16.372 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:16.372 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-03-29T10:41:16.372 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:16.372 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 145 k 2024-03-29T10:41:16.373 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 436 k 2024-03-29T10:41:16.373 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:41:16.603 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 627 kB/s | 145 kB 00:00 2024-03-29T10:41:16.604 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:16.605 INFO:teuthology.orchestra.run.smithi189.stdout:Total 623 kB/s | 145 kB 00:00 2024-03-29T10:41:16.605 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:41:16.612 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:41:16.612 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:41:16.666 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:41:16.666 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:41:16.806 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:41:16.870 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:41:16.895 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:41:16.895 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:41:16.895 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:41:16.895 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:17.847 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:41:17.848 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:17.848 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:41:17.848 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:41:17.848 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:17.848 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:18.090 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr 2024-03-29T10:41:18.669 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:18.771 INFO:teuthology.orchestra.run.smithi189.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:41:18.823 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:18.824 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-03-29T10:41:18.824 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:18.890 INFO:teuthology.orchestra.run.smithi098.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 4.4 kB/s | 32 kB 00:07 2024-03-29T10:41:18.895 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:41:19.157 INFO:teuthology.orchestra.run.smithi098.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 3.8 MB/s | 1.0 MB 00:00 2024-03-29T10:41:19.240 INFO:teuthology.orchestra.run.smithi098.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 711 kB/s | 59 kB 00:00 2024-03-29T10:41:19.316 INFO:teuthology.orchestra.run.smithi098.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-29T10:41:19.454 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:19.605 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:19.606 INFO:teuthology.orchestra.run.smithi189.stdout:====================================================================================== 2024-03-29T10:41:19.606 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-03-29T10:41:19.606 INFO:teuthology.orchestra.run.smithi189.stdout:====================================================================================== 2024-03-29T10:41:19.606 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:41:19.606 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:41:19.606 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-29T10:41:19.606 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:41:19.606 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout:====================================================================================== 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout:Install 12 Packages 2024-03-29T10:41:19.607 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:19.608 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 6.9 M 2024-03-29T10:41:19.608 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 26 M 2024-03-29T10:41:19.608 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:41:20.114 INFO:teuthology.orchestra.run.smithi189.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 170 kB/s | 31 kB 00:00 2024-03-29T10:41:20.140 INFO:teuthology.orchestra.run.smithi189.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 73 kB/s | 15 kB 00:00 2024-03-29T10:41:20.298 INFO:teuthology.orchestra.run.smithi189.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-29T10:41:20.332 INFO:teuthology.orchestra.run.smithi189.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-29T10:41:20.374 INFO:teuthology.orchestra.run.smithi189.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.0 MB/s | 1.8 MB 00:00 2024-03-29T10:41:20.583 INFO:teuthology.orchestra.run.smithi189.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.0 MB/s | 863 kB 00:00 2024-03-29T10:41:20.683 INFO:teuthology.orchestra.run.smithi189.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 63 kB/s | 19 kB 00:00 2024-03-29T10:41:20.833 INFO:teuthology.orchestra.run.smithi189.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T10:41:20.908 INFO:teuthology.orchestra.run.smithi189.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 641 kB/s | 144 kB 00:00 2024-03-29T10:41:20.933 INFO:teuthology.orchestra.run.smithi189.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 307 kB/s | 31 kB 00:00 2024-03-29T10:41:21.051 INFO:teuthology.orchestra.run.smithi189.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.3 MB/s | 188 kB 00:00 2024-03-29T10:41:21.184 INFO:teuthology.orchestra.run.smithi189.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.4 MB/s | 2.0 MB 00:00 2024-03-29T10:41:21.189 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:21.189 INFO:teuthology.orchestra.run.smithi189.stdout:Total 4.3 MB/s | 6.9 MB 00:01 2024-03-29T10:41:21.206 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:41:21.223 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:41:21.223 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:41:21.400 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:41:21.400 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:41:21.905 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:41:21.976 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:41:22.019 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:41:22.065 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:41:22.105 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:41:22.202 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:41:22.338 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:41:22.431 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:41:22.493 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:41:22.521 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:41:22.553 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:41:22.793 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:41:22.836 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:41:23.676 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:41:23.676 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:41:23.676 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:41:23.676 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:41:23.676 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:41:23.676 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:41:23.676 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:41:23.676 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:41:23.677 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:41:23.677 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:41:23.677 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:41:23.677 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:41:24.040 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:41:24.040 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:24.040 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:41:24.040 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:41:24.040 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:41:24.040 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:41:24.040 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:41:24.040 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:41:24.041 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:41:24.041 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:41:24.041 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:41:24.041 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:41:24.041 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:41:24.041 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:41:24.041 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:41:24.041 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:24.041 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:24.265 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:41:24.864 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:25.024 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:25.025 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================================ 2024-03-29T10:41:25.025 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-03-29T10:41:25.025 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================================ 2024-03-29T10:41:25.025 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:41:25.026 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:25.027 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:41:25.027 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================================ 2024-03-29T10:41:25.027 INFO:teuthology.orchestra.run.smithi189.stdout:Install 12 Packages 2024-03-29T10:41:25.027 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:25.027 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 37 M 2024-03-29T10:41:25.027 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 209 M 2024-03-29T10:41:25.027 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:41:25.708 INFO:teuthology.orchestra.run.smithi189.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 231 kB/s | 32 kB 00:00 2024-03-29T10:41:25.741 INFO:teuthology.orchestra.run.smithi189.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 538 kB/s | 17 kB 00:00 2024-03-29T10:41:25.825 INFO:teuthology.orchestra.run.smithi189.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-29T10:41:25.850 INFO:teuthology.orchestra.run.smithi189.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 319 kB/s | 34 kB 00:00 2024-03-29T10:41:26.117 INFO:teuthology.orchestra.run.smithi189.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-29T10:41:26.176 INFO:teuthology.orchestra.run.smithi189.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-03-29T10:41:26.201 INFO:teuthology.orchestra.run.smithi189.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-29T10:41:26.318 INFO:teuthology.orchestra.run.smithi189.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-14 9.9 MB/s | 7.4 MB 00:00 2024-03-29T10:41:26.593 INFO:teuthology.orchestra.run.smithi189.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-29T10:41:26.710 INFO:teuthology.orchestra.run.smithi189.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 501 kB/s | 195 kB 00:00 2024-03-29T10:41:26.893 INFO:teuthology.orchestra.run.smithi189.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-29T10:41:27.144 INFO:teuthology.orchestra.run.smithi189.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 446 kB/s | 245 kB 00:00 2024-03-29T10:41:27.145 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:27.146 INFO:teuthology.orchestra.run.smithi189.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-29T10:41:27.188 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:41:27.213 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:41:27.213 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:41:27.580 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:41:27.580 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:41:28.318 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:41:28.392 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:41:28.431 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:41:28.517 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:41:28.559 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:41:28.738 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:41:28.767 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:41:28.852 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:41:29.504 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:41:29.613 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:41:30.857 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:41:31.889 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:41:31.915 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:41:34.384 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:41:34.385 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:41:34.385 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:41:34.385 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:41:34.385 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:41:34.385 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:41:34.385 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:41:34.385 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:41:34.385 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:41:34.385 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:41:34.385 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:41:34.385 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:41:34.855 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:41:34.856 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:34.856 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:41:34.856 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:41:34.856 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:41:34.856 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:41:34.856 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:41:34.856 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:41:34.856 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:41:34.856 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:41:34.856 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:41:34.856 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:41:34.856 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:41:34.856 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:41:34.857 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:41:34.857 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:34.857 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:35.215 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:41:35.802 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:35.955 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:35.956 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================== 2024-03-29T10:41:35.956 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-03-29T10:41:35.957 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================== 2024-03-29T10:41:35.957 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:41:35.957 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:41:35.957 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-29T10:41:35.957 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:41:35.957 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:41:35.957 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:41:35.957 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:41:35.957 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:41:35.957 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:41:35.957 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:41:35.957 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:41:35.957 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:41:35.958 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:41:35.958 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:41:35.958 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:41:35.958 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:35.958 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:41:35.958 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================== 2024-03-29T10:41:35.958 INFO:teuthology.orchestra.run.smithi189.stdout:Install 13 Packages 2024-03-29T10:41:35.958 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:35.959 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 2.0 M 2024-03-29T10:41:35.959 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 26 M 2024-03-29T10:41:35.959 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:41:36.163 INFO:teuthology.orchestra.run.smithi189.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2024-03-29T10:41:36.182 INFO:teuthology.orchestra.run.smithi189.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 140 kB/s | 24 kB 00:00 2024-03-29T10:41:36.232 INFO:teuthology.orchestra.run.smithi189.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-29T10:41:36.257 INFO:teuthology.orchestra.run.smithi189.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:41:36.274 INFO:teuthology.orchestra.run.smithi189.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-29T10:41:36.300 INFO:teuthology.orchestra.run.smithi189.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T10:41:36.333 INFO:teuthology.orchestra.run.smithi189.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 160 kB/s | 52 kB 00:00 2024-03-29T10:41:36.450 INFO:teuthology.orchestra.run.smithi189.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 184 kB/s | 32 kB 00:00 2024-03-29T10:41:36.492 INFO:teuthology.orchestra.run.smithi189.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2024-03-29T10:41:36.601 INFO:teuthology.orchestra.run.smithi189.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 921 kB/s | 203 kB 00:00 2024-03-29T10:41:36.634 INFO:teuthology.orchestra.run.smithi189.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 5.6 MB/s | 1.0 MB 00:00 2024-03-29T10:41:36.659 INFO:teuthology.orchestra.run.smithi189.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-03-29T10:41:36.684 INFO:teuthology.orchestra.run.smithi189.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-29T10:41:36.688 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:36.689 INFO:teuthology.orchestra.run.smithi189.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-03-29T10:41:36.689 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:41:36.702 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:41:36.702 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:41:36.794 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:41:36.794 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:41:37.317 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:41:37.491 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:41:37.544 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:41:37.608 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:41:37.645 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:41:37.686 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:41:37.781 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:41:37.890 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:41:37.938 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:41:38.680 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:41:38.734 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:41:38.768 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:41:38.816 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:41:38.838 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:41:39.634 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:41:39.634 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:41:39.634 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:41:39.634 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:41:39.634 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:41:39.634 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:41:39.634 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:41:39.635 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:41:39.635 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:41:39.635 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:41:39.635 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:41:39.635 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:41:39.635 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:41:40.012 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:41:40.012 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:40.012 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:41:40.012 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:41:40.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:41:40.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:41:40.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:41:40.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:41:40.012 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:41:40.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:41:40.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:41:40.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:41:40.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:41:40.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:41:40.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:41:40.013 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:41:40.013 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:40.013 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:40.243 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:41:40.812 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:40.965 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:40.966 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================== 2024-03-29T10:41:40.966 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-03-29T10:41:40.966 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================== 2024-03-29T10:41:40.966 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:41:40.966 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:41:40.966 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-29T10:41:40.967 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:41:40.967 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:41:40.967 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:41:40.967 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:41:40.967 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:41:40.967 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:40.967 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:41:40.967 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================== 2024-03-29T10:41:40.967 INFO:teuthology.orchestra.run.smithi189.stdout:Install 6 Packages 2024-03-29T10:41:40.967 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:40.968 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 6.8 M 2024-03-29T10:41:40.968 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 33 M 2024-03-29T10:41:40.968 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:41:41.711 INFO:teuthology.orchestra.run.smithi189.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 441 kB/s | 138 kB 00:00 2024-03-29T10:41:41.735 INFO:teuthology.orchestra.run.smithi189.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 735 kB/s | 248 kB 00:00 2024-03-29T10:41:41.919 INFO:teuthology.orchestra.run.smithi189.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-29T10:41:42.086 INFO:teuthology.orchestra.run.smithi189.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2024-03-29T10:41:42.169 INFO:teuthology.orchestra.run.smithi189.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-29T10:41:42.203 INFO:teuthology.orchestra.run.smithi189.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 180 kB/s | 51 kB 00:00 2024-03-29T10:41:42.206 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:42.206 INFO:teuthology.orchestra.run.smithi189.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-03-29T10:41:42.219 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:41:42.230 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:41:42.230 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:41:42.318 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:41:42.319 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:41:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:41:42.788 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:41:42.846 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:41:43.357 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:41:43.441 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:41:43.488 INFO:teuthology.orchestra.run.smithi098.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 6.4 kB/s | 203 kB 00:31 2024-03-29T10:41:43.502 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:41:43.525 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:41:44.331 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:41:44.331 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:41:44.331 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:41:44.331 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:41:44.331 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:41:44.331 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:41:44.634 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:41:44.634 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:44.634 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:41:44.634 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:41:44.634 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:41:44.634 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:41:44.634 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:41:44.635 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:41:44.635 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:41:44.635 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:44.635 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:44.876 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-fuse 2024-03-29T10:41:45.442 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:45.594 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:45.594 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:45.594 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-03-29T10:41:45.594 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:45.595 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:41:45.595 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:41:45.595 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-29T10:41:45.595 INFO:teuthology.orchestra.run.smithi189.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:41:45.595 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:45.595 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:41:45.595 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:45.595 INFO:teuthology.orchestra.run.smithi189.stdout:Install 2 Packages 2024-03-29T10:41:45.595 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:45.596 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 964 k 2024-03-29T10:41:45.596 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 2.8 M 2024-03-29T10:41:45.596 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:41:45.737 INFO:teuthology.orchestra.run.smithi189.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 556 kB/s | 79 kB 00:00 2024-03-29T10:41:46.146 INFO:teuthology.orchestra.run.smithi189.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.6 MB/s | 885 kB 00:00 2024-03-29T10:41:46.147 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:46.147 INFO:teuthology.orchestra.run.smithi189.stdout:Total 1.7 MB/s | 964 kB 00:00 2024-03-29T10:41:46.147 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:41:46.168 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:41:46.168 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:41:46.242 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:41:46.242 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:41:46.428 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:41:46.572 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:41:46.638 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:41:47.419 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:41:47.420 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:41:47.708 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:41:47.708 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:47.709 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:41:47.709 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:41:47.709 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:47.709 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:47.974 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-volume 2024-03-29T10:41:48.553 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:48.709 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:48.710 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:48.710 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-03-29T10:41:48.710 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:48.710 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:41:48.710 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:41:48.710 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-29T10:41:48.710 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:41:48.710 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:48.711 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:41:48.711 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:48.711 INFO:teuthology.orchestra.run.smithi189.stdout:Install 2 Packages 2024-03-29T10:41:48.711 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:48.711 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 332 k 2024-03-29T10:41:48.711 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 1.5 M 2024-03-29T10:41:48.711 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:41:48.827 INFO:teuthology.orchestra.run.smithi189.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 597 kB/s | 69 kB 00:00 2024-03-29T10:41:48.952 INFO:teuthology.orchestra.run.smithi189.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 1.1 MB/s | 264 kB 00:00 2024-03-29T10:41:48.953 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:48.953 INFO:teuthology.orchestra.run.smithi189.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-29T10:41:48.953 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:41:48.962 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:41:48.962 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:41:49.006 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:41:49.006 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:41:49.155 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:41:49.334 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:41:49.371 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:41:49.387 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:41:49.387 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:41:49.388 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:41:49.388 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:50.093 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:41:50.450 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:41:50.451 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:50.451 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:41:50.451 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:41:50.451 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:41:50.451 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:50.451 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:50.767 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install librados-devel 2024-03-29T10:41:51.349 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:51.501 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:51.502 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:51.502 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repo Size 2024-03-29T10:41:51.502 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:51.502 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:41:51.502 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:41:51.503 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:51.503 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:41:51.503 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:51.503 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-03-29T10:41:51.503 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:51.503 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 127 k 2024-03-29T10:41:51.503 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 456 k 2024-03-29T10:41:51.503 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:41:51.716 INFO:teuthology.orchestra.run.smithi189.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 595 kB/s | 127 kB 00:00 2024-03-29T10:41:51.717 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:51.717 INFO:teuthology.orchestra.run.smithi189.stdout:Total 591 kB/s | 127 kB 00:00 2024-03-29T10:41:51.717 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:41:51.722 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:41:51.723 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:41:51.779 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:41:51.779 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:41:51.926 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:41:52.031 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:41:52.472 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:41:52.800 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:41:52.800 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:52.800 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:41:52.800 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:41:52.800 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:52.800 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:53.067 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install libcephfs2 2024-03-29T10:41:53.659 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:53.769 INFO:teuthology.orchestra.run.smithi189.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:41:53.826 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:53.827 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-03-29T10:41:53.827 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:53.889 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install libcephfs-devel 2024-03-29T10:41:54.483 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:54.642 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:54.643 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:54.643 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repo Size 2024-03-29T10:41:54.643 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:54.643 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:41:54.643 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:41:54.644 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:54.644 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:41:54.644 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:41:54.644 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-03-29T10:41:54.644 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:54.644 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 31 k 2024-03-29T10:41:54.644 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 140 k 2024-03-29T10:41:54.644 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:41:54.922 INFO:teuthology.orchestra.run.smithi189.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 112 kB/s | 31 kB 00:00 2024-03-29T10:41:54.923 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:41:54.923 INFO:teuthology.orchestra.run.smithi189.stdout:Total 112 kB/s | 31 kB 00:00 2024-03-29T10:41:54.923 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:41:54.927 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:41:54.927 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:41:54.950 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:41:54.950 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:41:55.069 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:41:55.193 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:41:55.603 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:41:55.866 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:41:55.867 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:55.867 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:41:55.867 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:41:55.867 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:41:55.867 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:56.037 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install librados2 2024-03-29T10:41:56.599 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:56.697 INFO:teuthology.orchestra.run.smithi189.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:41:56.750 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:56.751 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-03-29T10:41:56.751 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:56.804 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install librbd1 2024-03-29T10:41:57.370 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:57.469 INFO:teuthology.orchestra.run.smithi189.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:41:57.520 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:57.521 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-03-29T10:41:57.521 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:57.577 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-rados 2024-03-29T10:41:58.147 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:58.245 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:41:58.297 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:58.298 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-03-29T10:41:58.298 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:58.344 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-rgw 2024-03-29T10:41:58.909 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:59.007 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:41:59.058 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:59.059 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-03-29T10:41:59.059 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:59.109 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-cephfs 2024-03-29T10:41:59.682 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:41:59.779 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:41:59.831 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:41:59.832 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-03-29T10:41:59.832 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:41:59.882 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install python3-rbd 2024-03-29T10:42:00.450 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:42:00.549 INFO:teuthology.orchestra.run.smithi189.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:42:00.601 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:42:00.602 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-03-29T10:42:00.602 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:42:00.651 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install rbd-fuse 2024-03-29T10:42:01.219 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:42:01.369 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:42:01.369 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:42:01.369 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-03-29T10:42:01.370 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:42:01.370 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:42:01.370 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:42:01.370 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:42:01.370 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:42:01.370 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:42:01.370 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-03-29T10:42:01.370 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:42:01.370 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 85 k 2024-03-29T10:42:01.370 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 231 k 2024-03-29T10:42:01.370 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:42:01.689 INFO:teuthology.orchestra.run.smithi189.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 268 kB/s | 85 kB 00:00 2024-03-29T10:42:01.689 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:42:01.689 INFO:teuthology.orchestra.run.smithi189.stdout:Total 267 kB/s | 85 kB 00:00 2024-03-29T10:42:01.690 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:42:01.696 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:42:01.697 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:42:01.753 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:42:01.753 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:42:01.889 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:42:01.995 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:42:02.389 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:42:02.723 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:42:02.723 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:42:02.723 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:42:02.723 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:42:02.723 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:42:02.723 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:42:02.960 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install rbd-mirror 2024-03-29T10:42:03.540 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:42:03.698 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:42:03.699 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:42:03.699 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repo Size 2024-03-29T10:42:03.699 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:42:03.699 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:42:03.699 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:42:03.699 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:42:03.699 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:42:03.699 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:42:03.699 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-03-29T10:42:03.699 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:42:03.699 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 3.1 M 2024-03-29T10:42:03.699 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 13 M 2024-03-29T10:42:03.699 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:42:04.117 INFO:teuthology.orchestra.run.smithi189.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 7.4 MB/s | 3.1 MB 00:00 2024-03-29T10:42:04.117 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:42:04.118 INFO:teuthology.orchestra.run.smithi189.stdout:Total 7.4 MB/s | 3.1 MB 00:00 2024-03-29T10:42:04.118 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:42:04.127 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:42:04.127 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:42:04.200 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:42:04.200 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:42:04.649 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:42:04.718 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:42:04.742 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:42:04.742 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:42:04.742 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:42:04.742 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:42:04.742 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:42:04.742 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:42:05.861 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:42:05.861 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:42:05.862 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:42:05.862 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:42:05.862 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:42:05.862 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:42:06.078 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install rbd-nbd 2024-03-29T10:42:06.664 INFO:teuthology.orchestra.run.smithi189.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:40:18 AM UTC. 2024-03-29T10:42:06.815 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-29T10:42:06.816 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:42:06.816 INFO:teuthology.orchestra.run.smithi189.stdout: Package Architecture Version Repository Size 2024-03-29T10:42:06.816 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:42:06.816 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-29T10:42:06.816 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:42:06.816 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:42:06.816 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-29T10:42:06.816 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-03-29T10:42:06.817 INFO:teuthology.orchestra.run.smithi189.stdout:Install 1 Package 2024-03-29T10:42:06.817 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:42:06.817 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 171 k 2024-03-29T10:42:06.817 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 494 k 2024-03-29T10:42:06.817 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-29T10:42:07.120 INFO:teuthology.orchestra.run.smithi189.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 566 kB/s | 171 kB 00:00 2024-03-29T10:42:07.121 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:42:07.121 INFO:teuthology.orchestra.run.smithi189.stdout:Total 563 kB/s | 171 kB 00:00 2024-03-29T10:42:07.121 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-29T10:42:07.128 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-29T10:42:07.129 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-29T10:42:07.184 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-29T10:42:07.184 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-29T10:42:07.339 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-29T10:42:07.443 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:42:07.878 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:42:08.158 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:42:08.159 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:42:08.159 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-29T10:42:08.159 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:42:08.159 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:42:08.159 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-29T10:42:08.386 DEBUG:teuthology.parallel:result is None 2024-03-29T10:43:21.060 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] python3-certifi-2021.10.8-2.el9.noarch.rpm: Curl error (28): Timeout was reached for http://epel.stl.us.ssimn.org/9/Everything/x86_64/Packages/p/python3-certifi-2021.10.8-2.el9.noarch.rpm [Failed to connect to epel.stl.us.ssimn.org port 80: Connection timed out] 2024-03-29T10:43:21.459 INFO:teuthology.orchestra.run.smithi098.stdout:(13/13): python3-certifi-2021.10.8-2.el9.noarch 118 B/s | 15 kB 02:09 2024-03-29T10:43:21.460 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:21.460 INFO:teuthology.orchestra.run.smithi098.stdout:Total 16 kB/s | 2.0 MB 02:10 2024-03-29T10:43:21.460 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:43:21.475 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:43:21.475 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:43:21.572 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:43:21.572 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:43:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:43:22.244 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:43:22.303 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:43:22.358 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:43:22.388 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:43:22.429 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:43:22.521 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:43:22.629 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:43:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:43:23.390 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:43:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:43:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:43:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:43:23.549 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:43:24.279 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:43:24.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:43:24.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:43:24.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:43:24.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:43:24.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:43:24.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:43:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:43:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:43:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:43:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:43:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:43:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:43:24.649 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:43:24.649 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:24.649 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:43:24.649 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:43:24.649 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:43:24.649 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:43:24.649 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:43:24.649 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:43:24.649 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:43:24.649 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:43:24.649 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:43:24.650 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:43:24.650 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:43:24.650 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:43:24.650 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:43:24.650 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:43:24.650 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:24.650 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:24.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:43:25.463 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:33 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:25.616 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-29T10:43:25.617 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-03-29T10:43:25.618 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:25.618 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 6.8 M 2024-03-29T10:43:25.618 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 33 M 2024-03-29T10:43:25.618 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:43:26.114 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T10:43:26.155 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 448 kB/s | 138 kB 00:00 2024-03-29T10:43:26.206 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 693 kB/s | 248 kB 00:00 2024-03-29T10:43:26.348 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 348 kB/s | 58 kB 00:00 2024-03-29T10:43:26.373 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-03-29T10:43:26.389 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 320 kB/s | 51 kB 00:00 2024-03-29T10:43:26.393 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:26.393 INFO:teuthology.orchestra.run.smithi098.stdout:Total 8.8 MB/s | 6.8 MB 00:00 2024-03-29T10:43:26.406 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:43:26.417 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:43:26.417 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:43:26.504 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:43:26.505 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:43:26.796 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:43:26.954 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:43:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:43:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:43:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:43:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:43:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:43:28.486 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:43:28.487 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:43:28.487 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:43:28.487 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:43:28.487 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:43:28.487 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:43:28.797 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:43:28.798 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:28.798 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:43:28.798 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:43:28.798 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:43:28.798 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:43:28.798 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:43:28.798 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:43:28.798 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:43:28.798 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:28.798 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:29.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-fuse 2024-03-29T10:43:29.597 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:37 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:29.746 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:29.747 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:29.747 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T10:43:29.747 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:29.747 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:43:29.747 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:43:29.747 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T10:43:29.747 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:43:29.747 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:29.747 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:43:29.747 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:29.747 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-03-29T10:43:29.747 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:29.748 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 964 k 2024-03-29T10:43:29.748 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.8 M 2024-03-29T10:43:29.748 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:43:29.892 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 550 kB/s | 79 kB 00:00 2024-03-29T10:43:30.775 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 862 kB/s | 885 kB 00:01 2024-03-29T10:43:30.776 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:30.777 INFO:teuthology.orchestra.run.smithi098.stdout:Total 937 kB/s | 964 kB 00:01 2024-03-29T10:43:30.777 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:43:30.799 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:43:30.799 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:43:30.874 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:43:30.874 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:43:31.068 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:43:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:43:31.283 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:43:31.944 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:43:31.944 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:43:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:43:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:32.242 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:43:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:43:32.242 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:32.242 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:32.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-volume 2024-03-29T10:43:33.119 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:41 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:33.270 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:33.271 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-29T10:43:33.271 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:33.271 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:43:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:43:33.271 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-29T10:43:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:43:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:33.271 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:43:33.272 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:33.272 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-03-29T10:43:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:33.272 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 332 k 2024-03-29T10:43:33.272 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.5 M 2024-03-29T10:43:33.272 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:43:33.397 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 552 kB/s | 69 kB 00:00 2024-03-29T10:43:33.547 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 959 kB/s | 264 kB 00:00 2024-03-29T10:43:33.548 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:33.548 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T10:43:33.549 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:43:33.557 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:43:33.558 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:43:33.592 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:43:33.592 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:43:33.745 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:43:33.937 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:43:33.967 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:43:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:43:33.983 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:43:33.984 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:43:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:34.692 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:43:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:43:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:35.041 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:43:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:43:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:43:35.041 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:35.041 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:35.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados-devel 2024-03-29T10:43:35.962 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:43 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:36.114 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-29T10:43:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:43:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:43:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:43:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T10:43:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 127 k 2024-03-29T10:43:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 456 k 2024-03-29T10:43:36.115 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:43:36.346 INFO:teuthology.orchestra.run.smithi098.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 550 kB/s | 127 kB 00:00 2024-03-29T10:43:36.347 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:36.347 INFO:teuthology.orchestra.run.smithi098.stdout:Total 546 kB/s | 127 kB 00:00 2024-03-29T10:43:36.347 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:43:36.352 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:43:36.352 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:43:36.412 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:43:36.412 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:43:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:43:36.661 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:37.415 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:37.415 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:37.415 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:43:37.415 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:37.415 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:37.415 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:37.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs2 2024-03-29T10:43:38.256 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:46 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:38.359 INFO:teuthology.orchestra.run.smithi098.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:43:38.415 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:38.416 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T10:43:38.416 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:38.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs-devel 2024-03-29T10:43:39.043 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:47 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:39.194 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:39.195 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:39.195 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-29T10:43:39.195 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:39.195 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:43:39.195 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:43:39.195 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:39.195 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:43:39.195 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:39.195 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T10:43:39.195 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:39.196 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 31 k 2024-03-29T10:43:39.196 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 140 k 2024-03-29T10:43:39.196 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:43:39.357 INFO:teuthology.orchestra.run.smithi098.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 193 kB/s | 31 kB 00:00 2024-03-29T10:43:39.358 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:39.358 INFO:teuthology.orchestra.run.smithi098.stdout:Total 192 kB/s | 31 kB 00:00 2024-03-29T10:43:39.358 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:43:39.362 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:43:39.362 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:43:39.385 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:43:39.385 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:43:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:43:39.606 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:39.991 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:40.256 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:40.256 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:40.256 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:43:40.256 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:40.256 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:40.256 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:40.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados2 2024-03-29T10:43:40.979 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:48 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:41.076 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:43:41.127 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:41.128 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T10:43:41.128 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:41.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librbd1 2024-03-29T10:43:41.744 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:49 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:41.841 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:43:41.892 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:41.893 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T10:43:41.893 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:41.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rados 2024-03-29T10:43:42.513 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:50 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:42.612 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:43:42.664 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:42.665 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T10:43:42.665 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:42.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rgw 2024-03-29T10:43:43.293 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:51 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:43.390 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:43:43.442 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:43.443 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T10:43:43.443 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:43.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-cephfs 2024-03-29T10:43:44.066 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:52 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:44.163 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:43:44.215 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:44.216 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T10:43:44.216 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:44.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rbd 2024-03-29T10:43:44.835 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:52 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:44.932 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:43:44.984 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:44.985 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-29T10:43:44.985 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:45.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-fuse 2024-03-29T10:43:45.608 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:53 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:45.756 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:45.757 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:45.757 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-29T10:43:45.757 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:45.757 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:43:45.758 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:43:45.758 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:45.758 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:43:45.758 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:45.758 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T10:43:45.758 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:45.758 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 85 k 2024-03-29T10:43:45.758 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 231 k 2024-03-29T10:43:45.758 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:43:46.069 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 274 kB/s | 85 kB 00:00 2024-03-29T10:43:46.070 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:46.070 INFO:teuthology.orchestra.run.smithi098.stdout:Total 273 kB/s | 85 kB 00:00 2024-03-29T10:43:46.070 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:43:46.077 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:43:46.077 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:43:46.132 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:43:46.132 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:43:46.267 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:43:46.369 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:46.764 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:47.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:47.065 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:47.065 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:43:47.065 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:47.065 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:47.065 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:47.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-mirror 2024-03-29T10:43:47.868 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:55 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:48.021 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:48.022 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:48.022 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-29T10:43:48.022 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:48.022 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:43:48.022 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:43:48.022 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:48.022 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:43:48.022 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:48.022 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T10:43:48.022 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:48.023 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 3.1 M 2024-03-29T10:43:48.023 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 13 M 2024-03-29T10:43:48.023 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:43:48.398 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 8.2 MB/s | 3.1 MB 00:00 2024-03-29T10:43:48.399 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:48.399 INFO:teuthology.orchestra.run.smithi098.stdout:Total 8.2 MB/s | 3.1 MB 00:00 2024-03-29T10:43:48.399 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:43:48.408 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:43:48.408 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:43:48.481 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:43:48.482 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:43:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:43:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:48.990 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:43:48.990 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:43:48.991 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:43:48.991 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:43:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:50.114 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:50.114 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:50.114 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:43:50.114 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:50.114 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:50.115 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:50.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-nbd 2024-03-29T10:43:50.954 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:58 ago on Fri 29 Mar 2024 10:39:52 AM UTC. 2024-03-29T10:43:51.103 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-29T10:43:51.103 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-29T10:43:51.103 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:51.103 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-29T10:43:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:43:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:51.104 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-29T10:43:51.104 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-29T10:43:51.104 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-29T10:43:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:51.104 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 171 k 2024-03-29T10:43:51.104 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 494 k 2024-03-29T10:43:51.104 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-29T10:43:51.394 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 594 kB/s | 171 kB 00:00 2024-03-29T10:43:51.394 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:43:51.394 INFO:teuthology.orchestra.run.smithi098.stdout:Total 591 kB/s | 171 kB 00:00 2024-03-29T10:43:51.395 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-29T10:43:51.402 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-29T10:43:51.402 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-29T10:43:51.458 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-29T10:43:51.458 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-29T10:43:51.613 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-29T10:43:51.718 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:52.161 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:52.446 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:43:52.446 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:52.446 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-29T10:43:52.446 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:43:52.446 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:52.446 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-29T10:43:52.692 DEBUG:teuthology.parallel:result is None 2024-03-29T10:43:52.692 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:43:52.909 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:43:52.929 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:43:52.929 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:43:52.929 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:43:52.931 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:43:53.067 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:43:53.093 INFO:teuthology.orchestra.run.smithi189.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:43:53.093 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:43:53.093 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:43:53.095 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:43:53.095 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:43:53.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:43:53.122 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:43:53.122 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:43:53.166 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:43:53.166 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:43:53.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:43:53.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:43:53.260 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:43:53.260 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:43:53.291 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:43:53.360 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:43:53.361 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:43:53.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:43:53.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:43:53.474 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:43:53.474 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:43:53.505 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:43:53.575 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:43:53.575 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:43:53.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:43:53.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:43:53.669 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:43:53.669 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:43:53.696 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:43:53.764 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:43:53.848 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:43:53.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2024-03-29T10:43:53.850 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /var/log/ceph 2024-03-29T10:43:53.879 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:43:53.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:43:53.919 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:43:53.946 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:43:53.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:43:53.985 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:43:54.024 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:43:54.024 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:43:54.024 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622649', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOs/ndKlvjGdBzG5f1iS4hR4q57LTrq2aAzoPkLTTFv0bqaek2BKZukHVQyRNNKa+it/lpQSm/z9NtWS+GyXZ3U=', 'smithi189.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwqt2fg/uG9aD+S9vmfcYKVAE4utlnhEcAyqIpN44VF5msL/7BqUAgmNf3en7xQFd8nKxjiBt19L8OdGwTYbNM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:43:54.025 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:43:54.053 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:43:54.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:43:54.108 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:43:54.136 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:43:54.136 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:43:54.187 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:43:54.188 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-03-29T10:43:54.243 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:43:54.243 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:43:54.244 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T10:43:54.244 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:43:54.244 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:43:54.244 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-29 10:43:49.250761577 +0000 2024-03-29T10:43:54.244 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-29 10:39:06.994928162 +0000 2024-03-29T10:43:54.244 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-29 10:39:06.994928162 +0000 2024-03-29T10:43:54.244 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-29 10:39:06.994928162 +0000 2024-03-29T10:43:54.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:43:54.308 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-29T10:43:54.308 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-29T10:43:54.308 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.00018412 s, 2.8 MB/s 2024-03-29T10:43:54.309 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:43:54.365 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-03-29T10:43:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:43:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:43:54.420 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T10:43:54.420 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:43:54.420 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:43:54.420 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-29 10:43:49.251761564 +0000 2024-03-29T10:43:54.420 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-29 10:39:07.352923589 +0000 2024-03-29T10:43:54.420 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-29 10:39:07.352923589 +0000 2024-03-29T10:43:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-29 10:39:07.352923589 +0000 2024-03-29T10:43:54.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:43:54.484 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-29T10:43:54.485 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-29T10:43:54.485 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000195637 s, 2.6 MB/s 2024-03-29T10:43:54.486 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:43:54.540 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-03-29T10:43:54.595 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:43:54.595 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:43:54.595 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T10:43:54.595 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:43:54.595 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:43:54.595 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-29 10:43:49.251761564 +0000 2024-03-29T10:43:54.595 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-29 10:39:07.693919234 +0000 2024-03-29T10:43:54.595 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-29 10:39:07.693919234 +0000 2024-03-29T10:43:54.595 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-29 10:39:07.693919234 +0000 2024-03-29T10:43:54.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:43:54.660 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-29T10:43:54.660 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-29T10:43:54.660 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000196038 s, 2.6 MB/s 2024-03-29T10:43:54.661 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:43:54.717 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-03-29T10:43:54.772 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:43:54.772 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:43:54.772 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T10:43:54.772 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:43:54.773 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:43:54.773 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-29 10:43:49.251761564 +0000 2024-03-29T10:43:54.773 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-29 10:39:08.031914916 +0000 2024-03-29T10:43:54.773 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-29 10:39:08.031914916 +0000 2024-03-29T10:43:54.773 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-29 10:39:08.031914916 +0000 2024-03-29T10:43:54.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:43:54.837 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-29T10:43:54.838 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-29T10:43:54.838 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000202533 s, 2.5 MB/s 2024-03-29T10:43:54.839 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:43:54.894 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:43:54.895 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:43:54.895 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:43:54.911 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:43:54.911 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-03-29T10:43:54.966 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:43:54.967 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:43:54.967 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T10:43:54.967 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:43:54.967 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:43:54.967 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-29 10:42:05.001683928 +0000 2024-03-29T10:43:54.967 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-29 10:39:06.991478721 +0000 2024-03-29T10:43:54.967 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-29 10:39:06.991478721 +0000 2024-03-29T10:43:54.967 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-29 10:39:06.991478721 +0000 2024-03-29T10:43:54.967 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:43:55.033 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-29T10:43:55.033 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-29T10:43:55.033 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000282542 s, 1.8 MB/s 2024-03-29T10:43:55.034 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:43:55.091 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-03-29T10:43:55.145 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:43:55.146 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:43:55.146 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T10:43:55.146 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:43:55.146 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:43:55.146 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-29 10:42:05.001683928 +0000 2024-03-29T10:43:55.146 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-29 10:39:07.329472128 +0000 2024-03-29T10:43:55.146 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-29 10:39:07.329472128 +0000 2024-03-29T10:43:55.146 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-29 10:39:07.329472128 +0000 2024-03-29T10:43:55.146 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:43:55.211 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-29T10:43:55.211 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-29T10:43:55.211 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000267373 s, 1.9 MB/s 2024-03-29T10:43:55.213 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:43:55.268 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-03-29T10:43:55.322 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:43:55.322 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:43:55.322 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T10:43:55.322 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:43:55.322 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:43:55.322 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-29 10:42:05.001683928 +0000 2024-03-29T10:43:55.323 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-29 10:39:07.698464932 +0000 2024-03-29T10:43:55.323 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-29 10:39:07.698464932 +0000 2024-03-29T10:43:55.323 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-29 10:39:07.698464932 +0000 2024-03-29T10:43:55.323 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:43:55.388 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-29T10:43:55.388 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-29T10:43:55.388 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000222728 s, 2.3 MB/s 2024-03-29T10:43:55.389 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:43:55.445 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-03-29T10:43:55.500 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:43:55.500 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:43:55.501 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T10:43:55.501 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:43:55.501 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:43:55.501 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-29 10:42:05.002683909 +0000 2024-03-29T10:43:55.501 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-29 10:39:08.039458281 +0000 2024-03-29T10:43:55.501 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-29 10:39:08.039458281 +0000 2024-03-29T10:43:55.501 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-29 10:39:08.039458281 +0000 2024-03-29T10:43:55.502 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:43:55.566 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-29T10:43:55.566 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-29T10:43:55.566 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000204521 s, 2.5 MB/s 2024-03-29T10:43:55.567 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:43:55.623 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:43:55.623 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:43:55.623 INFO:tasks.ceph:Generating config... 2024-03-29T10:43:55.624 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:43:55.624 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:43:55.625 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:43:55.625 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:43:55.625 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:43:55.625 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:43:55.625 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:43:55.625 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:43:55.625 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:43:55.625 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:43:55.625 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:43:55.625 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:43:55.625 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:43:55.625 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:43:55.625 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:43:55.625 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:43:55.626 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:43:55.626 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:43:55.626 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:43:55.626 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:43:55.626 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:43:55.626 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:43:55.626 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:43:55.626 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:43:55.626 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:43:55.626 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:43:55.626 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:43:55.626 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:43:55.626 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:43:55.626 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:43:55.627 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:43:55.628 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:43:55.628 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:43:55.628 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:43:55.628 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:43:55.628 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:43:55.628 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:43:55.690 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:43:55.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:43:55.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:43:55.763 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.98:6789]'), ('mon.c', '[v1:172.21.15.98:6790]'), ('mon.b', '[v1:172.21.15.189:6789]')] 2024-03-29T10:43:55.764 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.98:6789],[v1:172.21.15.98:6790],[v1:172.21.15.189:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:43:55.764 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:43:55.764 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:43:55.818 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.98:6789]' --addv c '[v1:172.21.15.98:6790]' --addv b '[v1:172.21.15.189:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:43:55.902 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:43:55.903 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: generated fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:43:55.903 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = quincy 2024-03-29T10:43:55.903 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 0 2024-03-29T10:43:55.903 INFO:teuthology.orchestra.run.smithi098.stdout:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:43:55.903 INFO:teuthology.orchestra.run.smithi098.stdout:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:43:55.903 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:43:55.903 INFO:teuthology.orchestra.run.smithi098.stdout:min_mon_release 17 (quincy) 2024-03-29T10:43:55.903 INFO:teuthology.orchestra.run.smithi098.stdout:election_strategy: 1 2024-03-29T10:43:55.903 INFO:teuthology.orchestra.run.smithi098.stdout:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:43:55.903 INFO:teuthology.orchestra.run.smithi098.stdout:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:43:55.903 INFO:teuthology.orchestra.run.smithi098.stdout:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:43:55.904 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:43:55.905 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:43:55.919 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 556f8916-2d97-4161-a370-2566dd79bd5f... 2024-03-29T10:43:55.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:43:55.962 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:43:56.009 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-03-29T10:43:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2024-03-29T10:43:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:43:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2024-03-29T10:43:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2024-03-29T10:43:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2024-03-29T10:43:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2024-03-29T10:43:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2024-03-29T10:43:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.010 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:43:56.011 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.012 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "[v1:172.21.15.98:6789],[v1:172.21.15.98:6790],[v1:172.21.15.189:6789]" 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: lockdep = True 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: mon client directed command retry = 5 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: mon election default strategy = 1 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: ms bind msgr2 = False 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject socket failures = 5000 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: ms type = async 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.013 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2024-03-29T10:43:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2024-03-29T10:43:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:43:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2024-03-29T10:43:56.014 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2024-03-29T10:43:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2024-03-29T10:43:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2024-03-29T10:43:56.015 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2024-03-29T10:43:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug trim objects = true 2024-03-29T10:43:56.016 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2024-03-29T10:43:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2024-03-29T10:43:56.017 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:43:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2024-03-29T10:43:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.018 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2024-03-29T10:43:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:43:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2024-03-29T10:43:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2024-03-29T10:43:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2024-03-29T10:43:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2024-03-29T10:43:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore compression algorithm = zstd 2024-03-29T10:43:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore compression mode = aggressive 2024-03-29T10:43:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2024-03-29T10:43:56.019 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore zero block detection = True 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 20 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 20 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: debug monc = 20 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 10 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: osd class default list = * 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: osd class load list = * 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: osd client watch timeout = 120 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify cached snaps = True 2024-03-29T10:43:56.020 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify missing on start = True 2024-03-29T10:43:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:43:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock override recovery settings = True 2024-03-29T10:43:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2024-03-29T10:43:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: osd op queue = debug_random 2024-03-29T10:43:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: osd op queue cut off = debug_random 2024-03-29T10:43:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: osd shutdown pgref assert = True 2024-03-29T10:43:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.021 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2024-03-29T10:43:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-29T10:43:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2024-03-29T10:43:56.021 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:43:56.022 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:43:56.023 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:43:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: mon min osdmap epochs = 50 2024-03-29T10:43:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune interval = 2 2024-03-29T10:43:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune min = 15 2024-03-29T10:43:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:43:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: mon scrub interval = 300 2024-03-29T10:43:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on pool no app = False 2024-03-29T10:43:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: paxos service trim min = 10 2024-03-29T10:43:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:43:56.024 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2024-03-29T10:43:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2024-03-29T10:43:56.024 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2024-03-29T10:43:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2024-03-29T10:43:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:43:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:43:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: debug monc = 20 2024-03-29T10:43:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-29T10:43:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: debug objecter = 20 2024-03-29T10:43:56.025 INFO:teuthology.orchestra.run.smithi098.stdout: debug rados = 20 2024-03-29T10:43:56.025 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2024-03-29T10:43:56.025 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2024-03-29T10:43:56.025 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2024-03-29T10:43:56.026 INFO:teuthology.orchestra.run.smithi189.stdout:[global] 2024-03-29T10:43:56.026 INFO:teuthology.orchestra.run.smithi189.stdout: chdir = "" 2024-03-29T10:43:56.026 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:43:56.026 INFO:teuthology.orchestra.run.smithi189.stdout: auth supported = cephx 2024-03-29T10:43:56.026 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.026 INFO:teuthology.orchestra.run.smithi189.stdout: filestore xattr use omap = true 2024-03-29T10:43:56.026 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.026 INFO:teuthology.orchestra.run.smithi189.stdout: mon clock drift allowed = 1.000 2024-03-29T10:43:56.026 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: osd crush chooseleaf type = 0 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: auth debug = true 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on old message = true 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on bug = true 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: mon pg warn max object skew = 0 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default size = 2 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.027 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow primary affinity = true 2024-03-29T10:43:56.028 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow pg remap = true 2024-03-29T10:43:56.028 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:43:56.028 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:43:56.028 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on no sortbitwise = false 2024-03-29T10:43:56.028 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:43:56.028 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on too few osds = false 2024-03-29T10:43:56.028 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:43:56.028 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:43:56.028 INFO:teuthology.orchestra.run.smithi189.stdout: mon_allow_pool_size_one = true 2024-03-29T10:43:56.028 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.028 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: osd default data pool replay window = 5 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: mon allow pool delete = true 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: mon cluster log file level = debug 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: debug asserts on shutdown = true 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: mon health detail to clog = false 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: mon host = "[v1:172.21.15.98:6789],[v1:172.21.15.98:6790],[v1:172.21.15.189:6789]" 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: lockdep = True 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: mon client directed command retry = 5 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: mon election default strategy = 1 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: ms bind msgr2 = False 2024-03-29T10:43:56.029 INFO:teuthology.orchestra.run.smithi189.stdout: ms inject socket failures = 5000 2024-03-29T10:43:56.030 INFO:teuthology.orchestra.run.smithi189.stdout: ms type = async 2024-03-29T10:43:56.030 INFO:teuthology.orchestra.run.smithi189.stdout: fsid = 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:43:56.030 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.030 INFO:teuthology.orchestra.run.smithi189.stdout:[osd] 2024-03-29T10:43:56.030 INFO:teuthology.orchestra.run.smithi189.stdout: osd journal size = 100 2024-03-29T10:43:56.030 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.030 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub load threshold = 5.0 2024-03-29T10:43:56.030 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub max interval = 600 2024-03-29T10:43:56.030 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:43:56.030 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.030 INFO:teuthology.orchestra.run.smithi189.stdout: osd recover clone overlap = true 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: osd recovery max chunk = 1048576 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug shutdown = true 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug op order = true 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify stray on activate = true 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug trim objects = true 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: osd open classes on start = true 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug pg log writeout = true 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: osd map max advance = 10 2024-03-29T10:43:56.031 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: journal zero on create = true 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: filestore apply finisher threads = 3 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: bdev debug aio = true 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug misdirected ops = true 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore block size = 96636764160 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore compression algorithm = zstd 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore compression mode = aggressive 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore fsck on mount = True 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore zero block detection = True 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluefs = 20 2024-03-29T10:43:56.032 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluestore = 20 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: debug monc = 20 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: debug osd = 20 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: debug rocksdb = 10 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd full ratio = 0.9 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: osd class default list = * 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: osd class load list = * 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: osd client watch timeout = 120 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify cached snaps = True 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify missing on start = True 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:43:56.033 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock override recovery settings = True 2024-03-29T10:43:56.034 INFO:teuthology.orchestra.run.smithi189.stdout: osd objectstore = bluestore 2024-03-29T10:43:56.034 INFO:teuthology.orchestra.run.smithi189.stdout: osd op queue = debug_random 2024-03-29T10:43:56.034 INFO:teuthology.orchestra.run.smithi189.stdout: osd op queue cut off = debug_random 2024-03-29T10:43:56.034 INFO:teuthology.orchestra.run.smithi189.stdout: osd shutdown pgref assert = True 2024-03-29T10:43:56.034 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.034 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr] 2024-03-29T10:43:56.034 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-03-29T10:43:56.034 INFO:teuthology.orchestra.run.smithi189.stdout: debug mgr = 20 2024-03-29T10:43:56.034 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-03-29T10:43:56.034 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-03-29T10:43:56.034 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:43:56.034 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:43:56.034 INFO:teuthology.orchestra.run.smithi189.stdout: mgr/telemetry/nag = false 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout:[mon] 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: debug paxos = 20 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: mon data avail warn = 5 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: mon mgr mkfs grace = 240 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd reporter subtree level = osd 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd prime pg temp = true 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.035 INFO:teuthology.orchestra.run.smithi189.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: # 1m isn't quite enough 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: mon min osdmap epochs = 50 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: mon osdmap full prune interval = 2 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: mon osdmap full prune min = 15 2024-03-29T10:43:56.036 INFO:teuthology.orchestra.run.smithi189.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:43:56.037 INFO:teuthology.orchestra.run.smithi189.stdout: mon scrub interval = 300 2024-03-29T10:43:56.037 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on pool no app = False 2024-03-29T10:43:56.037 INFO:teuthology.orchestra.run.smithi189.stdout: paxos service trim min = 10 2024-03-29T10:43:56.037 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-29T10:43:56.037 INFO:teuthology.orchestra.run.smithi189.stdout:[client] 2024-03-29T10:43:56.037 INFO:teuthology.orchestra.run.smithi189.stdout: rgw cache enabled = true 2024-03-29T10:43:56.037 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable ops log = true 2024-03-29T10:43:56.037 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable usage log = true 2024-03-29T10:43:56.037 INFO:teuthology.orchestra.run.smithi189.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:43:56.037 INFO:teuthology.orchestra.run.smithi189.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:43:56.037 INFO:teuthology.orchestra.run.smithi189.stdout: debug monc = 20 2024-03-29T10:43:56.037 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-03-29T10:43:56.038 INFO:teuthology.orchestra.run.smithi189.stdout: debug objecter = 20 2024-03-29T10:43:56.038 INFO:teuthology.orchestra.run.smithi189.stdout: debug rados = 20 2024-03-29T10:43:56.038 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.a] 2024-03-29T10:43:56.038 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.c] 2024-03-29T10:43:56.038 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.b] 2024-03-29T10:43:56.040 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:43:56.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:43:56.133 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:43:56.134 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:43:56.134 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:43:56.148 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:43:56.148 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:43:56.203 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T10:43:56.203 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:43:56.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:43:56.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:43:56.282 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:43:56.282 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:43:56.337 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi189.front.sepia.ceph.com 2024-03-29T10:43:56.337 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:43:56.338 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:43:56.338 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:43:56.377 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:43:56.377 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:43:56.433 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:43:56.433 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:43:56.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:43:56.488 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:43:56.491 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:43:56.551 INFO:teuthology.orchestra.run.smithi189.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:43:56.554 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:43:56.554 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:43:56.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:43:56.596 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:43:56.611 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:43:56.653 INFO:teuthology.orchestra.run.smithi189.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:43:56.669 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:43:56.670 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:43:56.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:43:56.697 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:43:56.697 INFO:tasks.ceph:role: osd.0 2024-03-29T10:43:56.697 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T10:43:56.697 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:43:56.772 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:43:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:43:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:43:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:43:56.772 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:43:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:43:56.772 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:43:56.773 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:43:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:43:56.773 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:43:57.278 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-29T10:43:57.280 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-29T10:43:57.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:43:57.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:43:57.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:43:57.509 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:43:57.509 INFO:tasks.ceph:role: osd.1 2024-03-29T10:43:57.509 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T10:43:57.509 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:43:57.583 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:43:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:43:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:43:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:43:57.583 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:43:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:43:57.583 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:43:57.583 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:43:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:43:57.583 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:43:58.047 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-29T10:43:58.049 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-29T10:43:58.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:43:58.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:43:58.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:43:58.242 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:43:58.242 INFO:tasks.ceph:role: osd.2 2024-03-29T10:43:58.242 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T10:43:58.243 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:43:58.317 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:43:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:43:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:43:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:43:58.317 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:43:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:43:58.318 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:43:58.318 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:43:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:43:58.318 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:43:58.807 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-29T10:43:58.809 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-29T10:43:58.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:43:58.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:43:58.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:43:59.043 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:43:59.043 INFO:tasks.ceph:role: osd.3 2024-03-29T10:43:59.043 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T10:43:59.043 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:43:59.118 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:43:59.119 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:43:59.119 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:43:59.119 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:43:59.119 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:43:59.119 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:43:59.119 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:43:59.119 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:43:59.119 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:43:59.119 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:43:59.641 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-29T10:43:59.644 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-29T10:43:59.644 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:43:59.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:43:59.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:43:59.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:43:59.953+0000 7f1f3a4d9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:43:59.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:43:59.953+0000 7f1f3a4d9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:43:59.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:43:59.953+0000 7f1f3a4d9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:43:59.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:43:59.953+0000 7f1f3a4d9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:43:59.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:43:59.953+0000 7f1f3a4d9640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:43:59.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:43:59.954+0000 7f1f3a4d9640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:44:02.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:44:02.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:44:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:02.920+0000 7fc2f9414640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:44:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:02.920+0000 7fc2f9414640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:44:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:02.920+0000 7fc2f9414640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:44:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:02.920+0000 7fc2f9414640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:44:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:02.920+0000 7fc2f9414640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:44:02.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:02.921+0000 7fc2f9414640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:44:05.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:44:05.779 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:44:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:05.886+0000 7f4dca349640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:44:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:05.887+0000 7f4dca349640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:44:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:05.887+0000 7f4dca349640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:44:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:05.887+0000 7f4dca349640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:44:05.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:05.887+0000 7f4dca349640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:44:05.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:05.887+0000 7f4dca349640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:44:08.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:44:08.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:44:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:08.861+0000 7ff15c91b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:44:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:08.861+0000 7ff15c91b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:44:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:08.861+0000 7ff15c91b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:44:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:08.861+0000 7ff15c91b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:44:08.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:08.861+0000 7ff15c91b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:44:08.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:08.862+0000 7ff15c91b640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:44:11.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:44:11.735 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:44:11.767 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:44:11.767 INFO:tasks.ceph:role: osd.4 2024-03-29T10:44:11.767 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com 2024-03-29T10:44:11.767 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:44:11.846 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:44:11.847 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:44:11.847 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:44:11.847 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:44:11.847 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:44:11.847 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:44:11.847 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:44:11.847 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:44:11.847 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:44:11.847 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:44:12.371 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-03-29T10:44:12.373 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-03-29T10:44:12.374 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:44:12.494 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:44:12.531 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:44:12.601 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:44:12.601 INFO:tasks.ceph:role: osd.5 2024-03-29T10:44:12.601 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com 2024-03-29T10:44:12.601 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:44:12.678 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:44:12.678 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:44:12.678 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:44:12.678 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:44:12.678 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:44:12.678 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:44:12.678 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:44:12.678 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:44:12.678 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:44:12.678 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:44:13.128 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-03-29T10:44:13.130 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-03-29T10:44:13.130 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:44:13.226 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:44:13.257 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:44:13.325 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:44:13.325 INFO:tasks.ceph:role: osd.6 2024-03-29T10:44:13.325 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com 2024-03-29T10:44:13.326 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:44:13.400 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:44:13.400 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:44:13.400 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:44:13.400 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:44:13.400 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:44:13.400 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:44:13.400 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:44:13.400 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:44:13.400 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:44:13.401 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:44:13.890 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-03-29T10:44:13.892 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-03-29T10:44:13.893 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:44:14.040 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:44:14.071 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:44:14.137 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:44:14.137 INFO:tasks.ceph:role: osd.7 2024-03-29T10:44:14.138 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi189.front.sepia.ceph.com 2024-03-29T10:44:14.138 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:44:14.212 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:44:14.212 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:44:14.212 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:44:14.212 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:44:14.212 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:44:14.212 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:44:14.212 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:44:14.213 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:44:14.213 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:44:14.213 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:44:14.735 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-03-29T10:44:14.737 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-03-29T10:44:14.737 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:44:14.856 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:44:14.891 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:44:14.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:15.010+0000 7f704eb73640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:44:14.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:15.010+0000 7f704eb73640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:44:14.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:15.010+0000 7f704eb73640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:44:14.993 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:15.010+0000 7f704eb73640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:44:14.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:15.010+0000 7f704eb73640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:44:14.994 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:15.010+0000 7f704eb73640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:44:17.842 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:44:17.873 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:44:17.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:17.982+0000 7fd00019a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:44:17.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:17.983+0000 7fd00019a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:44:17.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:17.983+0000 7fd00019a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:44:17.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:17.983+0000 7fd00019a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:44:17.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:17.983+0000 7fd00019a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:44:17.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:17.983+0000 7fd00019a640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:44:20.810 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:44:20.854 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:44:20.949 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:20.966+0000 7ff2ce20d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:44:20.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:20.967+0000 7ff2ce20d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:44:20.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:20.967+0000 7ff2ce20d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:44:20.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:20.967+0000 7ff2ce20d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:44:20.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:20.967+0000 7ff2ce20d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:44:20.950 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:20.967+0000 7ff2ce20d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:44:23.797 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:44:23.828 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:44:23.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:23.938+0000 7f32cd6f5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:44:23.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:23.939+0000 7f32cd6f5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:44:23.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:23.939+0000 7f32cd6f5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:44:23.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:23.939+0000 7f32cd6f5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:44:23.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:23.939+0000 7f32cd6f5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:44:23.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:23.939+0000 7f32cd6f5640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:44:26.770 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:44:26.803 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:44:26.803 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:44:26.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:44:26.836 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:44:26.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:44:26.905 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:44:26.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:44:26.975 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:44:26.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:44:27.043 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:44:27.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:44:27.110 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:44:27.110 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:44:27.142 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:44:27.142 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:44:27.212 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:44:27.212 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:44:27.281 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:44:27.281 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:44:27.350 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:44:27.350 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:44:27.419 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:44:27.419 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:44:27.435 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:44:27.435 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:44:27.475 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:44:27.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:44:27.478 DEBUG:teuthology.orchestra.run.smithi189:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:44:27.519 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.y] 2024-03-29T10:44:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBsmwZmjsAxHhAAzqESVfO9Fn8erQM4HkD13A== 2024-03-29T10:44:27.519 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2024-03-29T10:44:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBvmwZmMsfeOBAAJsj74KklUq9FCSl8R8PZwQ== 2024-03-29T10:44:27.519 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2024-03-29T10:44:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBymwZm3gDnNhAA7W8H47xEuBRqEI9O+x164Q== 2024-03-29T10:44:27.519 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2024-03-29T10:44:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQB1mwZm2j/nNBAAn2/x5UsTCeAiSWrM8NAtFw== 2024-03-29T10:44:27.520 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.3] 2024-03-29T10:44:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQB4mwZmF5NiMxAARp0J/WxAqCBc7aUmj3RE9Q== 2024-03-29T10:44:27.520 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2024-03-29T10:44:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBsmwZmNdn2IRAAx35rmCNMc0LUsJaVEnLOJg== 2024-03-29T10:44:27.520 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.4] 2024-03-29T10:44:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQB/mwZm6OGoABAA/TualCWmzeBJI0w65dezcA== 2024-03-29T10:44:27.520 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.5] 2024-03-29T10:44:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCBmwZmEKSjOhAAt8DTkLA3zChaGBcakcQ5SQ== 2024-03-29T10:44:27.520 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.6] 2024-03-29T10:44:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCEmwZm7wuuORAAicVcct3PMeFBM4U7TdL78A== 2024-03-29T10:44:27.520 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.7] 2024-03-29T10:44:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCHmwZm+4gFOBAA3kFn38A7KlKvtYqKlTGjWw== 2024-03-29T10:44:27.520 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-03-29T10:44:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBsmwZmEDWkJBAAojej9E8PfKNNL4pR/AZUtA== 2024-03-29T10:44:27.521 INFO:teuthology.orchestra.run.smithi098.stdout:[client.1] 2024-03-29T10:44:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBsmwZmC2ENKBAA+MQ3C/59YrEbnLoqdS3DaA== 2024-03-29T10:44:27.542 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr.y] 2024-03-29T10:44:27.542 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBsmwZmjsAxHhAAzqESVfO9Fn8erQM4HkD13A== 2024-03-29T10:44:27.542 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.0] 2024-03-29T10:44:27.542 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBvmwZmMsfeOBAAJsj74KklUq9FCSl8R8PZwQ== 2024-03-29T10:44:27.542 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.1] 2024-03-29T10:44:27.542 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBymwZm3gDnNhAA7W8H47xEuBRqEI9O+x164Q== 2024-03-29T10:44:27.542 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.2] 2024-03-29T10:44:27.542 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQB1mwZm2j/nNBAAn2/x5UsTCeAiSWrM8NAtFw== 2024-03-29T10:44:27.543 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.3] 2024-03-29T10:44:27.543 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQB4mwZmF5NiMxAARp0J/WxAqCBc7aUmj3RE9Q== 2024-03-29T10:44:27.543 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr.x] 2024-03-29T10:44:27.543 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBsmwZmNdn2IRAAx35rmCNMc0LUsJaVEnLOJg== 2024-03-29T10:44:27.543 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.4] 2024-03-29T10:44:27.543 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQB/mwZm6OGoABAA/TualCWmzeBJI0w65dezcA== 2024-03-29T10:44:27.543 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.5] 2024-03-29T10:44:27.543 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQCBmwZmEKSjOhAAt8DTkLA3zChaGBcakcQ5SQ== 2024-03-29T10:44:27.543 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.6] 2024-03-29T10:44:27.543 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQCEmwZm7wuuORAAicVcct3PMeFBM4U7TdL78A== 2024-03-29T10:44:27.543 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.7] 2024-03-29T10:44:27.543 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQCHmwZm+4gFOBAA3kFn38A7KlKvtYqKlTGjWw== 2024-03-29T10:44:27.543 INFO:teuthology.orchestra.run.smithi189.stdout:[client.0] 2024-03-29T10:44:27.544 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBsmwZmEDWkJBAAojej9E8PfKNNL4pR/AZUtA== 2024-03-29T10:44:27.544 INFO:teuthology.orchestra.run.smithi189.stdout:[client.1] 2024-03-29T10:44:27.544 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBsmwZmC2ENKBAA+MQ3C/59YrEbnLoqdS3DaA== 2024-03-29T10:44:27.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:44:27.565 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:44:27.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:27.640 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:27.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:27.694 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:27.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:27.749 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:27.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:27.804 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:27.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:44:27.858 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:44:27.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:27.912 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:27.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:27.966 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:28.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:28.019 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:28.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:28.073 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:44:28.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:44:28.127 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:44:28.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:44:28.184 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:44:28.239 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:44:28.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:44:28.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:44:28.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:44:28.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:44:28.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:44:28.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:44:28.581 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:44:28.608 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:44:28.708 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:44:28.735 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:44:28.737 DEBUG:teuthology.orchestra.run.smithi189:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:44:28.791 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:44:28.791 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:44:28.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:44:28.793 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:44:28.794 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:44:28.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:44:28.796 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:44:28.796 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:44:28.796 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:44:28.833 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:44:28.833 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:44:28.833 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:44:28.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:44:28.836 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:44:28.836 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:44:28.836 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:44:28.838 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:44:28.838 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:44:28.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:44:28.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.957+0000 7f7049081640 10 monclient: get_monmap_and_config 2024-03-29T10:44:28.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.957+0000 7f7049081640 10 monclient: build_initial_monmap 2024-03-29T10:44:28.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.957+0000 7f7049081640 10 monclient: monmap: 2024-03-29T10:44:28.940 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-29T10:44:28.940 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:28.940 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T10:44:28.958052+0000 2024-03-29T10:44:28.940 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T10:44:28.958052+0000 2024-03-29T10:44:28.941 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:28.941 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:28.941 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:28.941 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:28.941 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:28.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:28.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.957+0000 7f7049081640 1 Processor -- start 2024-03-29T10:44:28.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.957+0000 7f7049081640 1 -- start start 2024-03-29T10:44:28.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.957+0000 7f7049081640 10 monclient: init 2024-03-29T10:44:28.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.958+0000 7f7049081640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:28.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.958+0000 7f7049081640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:28.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.958+0000 7f7049081640 10 monclient(hunting): picked mon.noname-c con 0x7f704409ac40 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:28.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.958+0000 7f7049081640 10 monclient(hunting): picked mon.noname-a con 0x7f704409c870 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:28.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.958+0000 7f7049081640 10 monclient(hunting): picked mon.noname-b con 0x7f70440b4360 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:28.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.958+0000 7f7049081640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704409dec0 con 0x7f704409c870 2024-03-29T10:44:28.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.958+0000 7f7049081640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704409da30 con 0x7f704409ac40 2024-03-29T10:44:28.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.958+0000 7f7049081640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70440b39c0 con 0x7f70440b4360 2024-03-29T10:44:28.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.958+0000 7f7049081640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:28.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.958+0000 7f7049081640 10 monclient(hunting): authenticate will time out at 889.155438s 2024-03-29T10:44:28.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.958+0000 7f704259c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f704409c870 0x7f704413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47136/0 (socket says 172.21.15.98:47136) 2024-03-29T10:44:28.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.958+0000 7f704259c640 1 -- 172.21.15.98:0/2196934926 learned_addr learned my addr 172.21.15.98:0/2196934926 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:44:28.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:28.958+0000 7f704359e640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f70440b4360 0x7f704414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:38182/0 (socket says 172.21.15.98:38182) 2024-03-29T10:44:31.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 10 monclient(hunting): tick 2024-03-29T10:44:31.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:44:31.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 1 monclient(hunting): continuing hunt 2024-03-29T10:44:31.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:44:31.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 1 -- 172.21.15.98:0/2196934926 >> v1:172.21.15.98:6790/0 conn(0x7f70440b4360 legacy=0x7f704414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:31.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 1 -- 172.21.15.98:0/2196934926 >> v1:172.21.15.189:6789/0 conn(0x7f704409ac40 legacy=0x7f70440b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:31.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 1 -- 172.21.15.98:0/2196934926 >> v1:172.21.15.98:6789/0 conn(0x7f704409c870 legacy=0x7f704413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:31.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:31.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 10 monclient(hunting): picked mon.noname-c con 0x7f70300036e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:31.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 10 monclient(hunting): picked mon.noname-b con 0x7f70300072d0 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:31.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 10 monclient(hunting): picked mon.noname-a con 0x7f703000b020 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:31.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 1 -- 172.21.15.98:0/2196934926 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70440b5160 con 0x7f703000b020 2024-03-29T10:44:31.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 1 -- 172.21.15.98:0/2196934926 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703000fe10 con 0x7f70300036e0 2024-03-29T10:44:31.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 1 -- 172.21.15.98:0/2196934926 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7030011050 con 0x7f70300072d0 2024-03-29T10:44:31.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:31.959+0000 7f7040d99640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:34.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 1 -- 172.21.15.98:0/2196934926 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037539461 0 0) 0x7f70440b5160 con 0x7f703000b020 2024-03-29T10:44:34.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:44:34.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 1 -- 172.21.15.98:0/2196934926 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f702c003520 con 0x7f703000b020 2024-03-29T10:44:34.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 1 -- 172.21.15.98:0/2196934926 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269152444 0 0) 0x7f7030011050 con 0x7f70300072d0 2024-03-29T10:44:34.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:44:34.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 1 -- 172.21.15.98:0/2196934926 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70440b5160 con 0x7f70300072d0 2024-03-29T10:44:34.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 1 -- 172.21.15.98:0/2196934926 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004641779 0 0) 0x7f703000fe10 con 0x7f70300036e0 2024-03-29T10:44:34.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient(hunting): my global_id is 4105 2024-03-29T10:44:34.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 1 -- 172.21.15.98:0/2196934926 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7030011050 con 0x7f70300036e0 2024-03-29T10:44:34.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 1 -- 172.21.15.98:0/2196934926 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3193936989 0 0) 0x7f70440b5160 con 0x7f70300072d0 2024-03-29T10:44:34.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 1 -- 172.21.15.98:0/2196934926 >> v1:172.21.15.189:6789/0 conn(0x7f70300036e0 legacy=0x7f7030005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 1 -- 172.21.15.98:0/2196934926 >> v1:172.21.15.98:6789/0 conn(0x7f703000b020 legacy=0x7f703000d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 1 monclient: found mon.noname-b 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 1 -- 172.21.15.98:0/2196934926 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7030012290 con 0x7f70300072d0 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient: _finish_auth 0 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 1 -- 172.21.15.98:0/2196934926 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f7038001470 con 0x7f70300072d0 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7041d9b640 10 monclient: dump: 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T10:44:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7049081640 5 monclient: authenticate success, global_id 4106 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.129+0000 7f7049081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.130+0000 7f7041d9b640 1 -- 172.21.15.98:0/2196934926 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70380015d0 con 0x7f70300072d0 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.130+0000 7f7041d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.130+0000 7f7049081640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.130+0000 7f7049081640 4 set_mon_vals no callback set 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.130+0000 7f7041d9b640 1 -- 172.21.15.98:0/2196934926 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f70380019e0 con 0x7f70300072d0 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.130+0000 7f7041d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.130+0000 7f7041d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.130+0000 7f7041d9b640 10 monclient: dump: 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.130+0000 7f7049081640 10 monclient: shutdown 2024-03-29T10:44:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.130+0000 7f7049081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.130+0000 7f7049081640 1 -- 172.21.15.98:0/2196934926 >> v1:172.21.15.98:6790/0 conn(0x7f70300072d0 legacy=0x7f7030009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.130+0000 7f7049081640 1 -- 172.21.15.98:0/2196934926 shutdown_connections 2024-03-29T10:44:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.130+0000 7f7049081640 1 -- 172.21.15.98:0/2196934926 >> 172.21.15.98:0/2196934926 conn(0x7f7044066b00 msgr2=0x7f7044066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.131+0000 7f7049081640 1 -- 172.21.15.98:0/2196934926 shutdown_connections 2024-03-29T10:44:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.131+0000 7f7049081640 1 -- 172.21.15.98:0/2196934926 wait complete. 2024-03-29T10:44:34.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.132+0000 7f7049081640 10 monclient: build_initial_monmap 2024-03-29T10:44:34.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.132+0000 7f7049081640 10 monclient: monmap: 2024-03-29T10:44:34.125 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-29T10:44:34.125 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:34.125 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-29T10:44:34.125 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-29T10:44:34.125 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:34.125 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:34.125 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:34.125 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:34.126 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.132+0000 7f7049081640 1 librados: starting msgr at 2024-03-29T10:44:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.132+0000 7f7049081640 1 librados: starting objecter 2024-03-29T10:44:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.133+0000 7f7049081640 1 Processor -- start 2024-03-29T10:44:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.133+0000 7f7049081640 1 -- start start 2024-03-29T10:44:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.133+0000 7f7049081640 1 librados: setting wanted keys 2024-03-29T10:44:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.133+0000 7f7049081640 1 librados: calling monclient init 2024-03-29T10:44:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.133+0000 7f7049081640 10 monclient: init 2024-03-29T10:44:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.133+0000 7f7049081640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.133+0000 7f7049081640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.133+0000 7f7049081640 10 monclient(hunting): picked mon.noname-c con 0x7f70300036e0 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.133+0000 7f7049081640 10 monclient(hunting): picked mon.noname-b con 0x7f70300072d0 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f7049081640 10 monclient(hunting): picked mon.noname-a con 0x7f703000b020 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f7049081640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70441d6de0 con 0x7f703000b020 2024-03-29T10:44:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f7049081640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70440b6ef0 con 0x7f70300072d0 2024-03-29T10:44:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f7042d9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f70300036e0 0x7f70440b5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:50582/0 (socket says 172.21.15.98:50582) 2024-03-29T10:44:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f7049081640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70440b71d0 con 0x7f70300036e0 2024-03-29T10:44:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f7042d9d640 1 -- 172.21.15.98:0/719268542 learned_addr learned my addr 172.21.15.98:0/719268542 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:44:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f7049081640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f7049081640 10 monclient(hunting): authenticate will time out at 894.331216s 2024-03-29T10:44:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 client.?.objecter ms_handle_connect 0x7f703000b020 2024-03-29T10:44:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 client.?.objecter ms_handle_connect 0x7f70300072d0 2024-03-29T10:44:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 client.?.objecter ms_handle_connect 0x7f70300036e0 2024-03-29T10:44:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091264491 0 0) 0x7f70441d6de0 con 0x7f703000b020 2024-03-29T10:44:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:44:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70300119a0 con 0x7f703000b020 2024-03-29T10:44:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3105212760 0 0) 0x7f70300119a0 con 0x7f703000b020 2024-03-29T10:44:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70441d6de0 con 0x7f703000b020 2024-03-29T10:44:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f703c001c70 con 0x7f703000b020 2024-03-29T10:44:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 monclient(hunting): dump: 2024-03-29T10:44:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T10:44:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.130 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.134+0000 7f702b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 625819652 0 0) 0x7f70441d6de0 con 0x7f703000b020 2024-03-29T10:44:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 >> v1:172.21.15.98:6790/0 conn(0x7f70300036e0 legacy=0x7f70440b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 >> v1:172.21.15.189:6789/0 conn(0x7f70300072d0 legacy=0x7f70441d7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 1 monclient: found mon.a 2024-03-29T10:44:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70440b75f0 con 0x7f703000b020 2024-03-29T10:44:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f7049081640 5 monclient: authenticate success, global_id 4113 2024-03-29T10:44:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f7049081640 10 monclient: _renew_subs 2024-03-29T10:44:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f7049081640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f7049081640 1 -- 172.21.15.98:0/719268542 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70441d7980 con 0x7f703000b020 2024-03-29T10:44:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f7049081640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f7049081640 10 monclient: _renew_subs 2024-03-29T10:44:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f7049081640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f7049081640 1 -- 172.21.15.98:0/719268542 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70441d7cd0 con 0x7f703000b020 2024-03-29T10:44:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f7049081640 1 librados: init done 2024-03-29T10:44:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f703c0020e0 con 0x7f703000b020 2024-03-29T10:44:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f703c004d00 con 0x7f703000b020 2024-03-29T10:44:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 10 monclient: dump: 2024-03-29T10:44:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T10:44:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.135 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f704159a640 4 set_mon_vals no callback set 2024-03-29T10:44:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f703c005270 con 0x7f703000b020 2024-03-29T10:44:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1536988769 0 0) 0x7f703c006830 con 0x7f703000b020 2024-03-29T10:44:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 10 client.4113.objecter ms_dispatch 0x7f7044000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:44:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:44:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:44:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.135+0000 7f702b7fe640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T10:44:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.136+0000 7f7049081640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.136+0000 7f7049081640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.136+0000 7f7049081640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.136+0000 7f7049081640 1 -- 172.21.15.98:0/719268542 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7024005080 con 0x7f703000b020 2024-03-29T10:44:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.140+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f703c006b10 con 0x7f703000b020 2024-03-29T10:44:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.140+0000 7f702b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.140+0000 7f702b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.213+0000 7f7049081640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:44:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.213+0000 7f7049081640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:44:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.213+0000 7f7049081640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:34.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.213+0000 7f7049081640 1 -- 172.21.15.98:0/719268542 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7024005330 con 0x7f703000b020 2024-03-29T10:44:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.214+0000 7f702b7fe640 1 -- 172.21.15.98:0/719268542 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f703c0058e0 con 0x7f703000b020 2024-03-29T10:44:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.214+0000 7f702b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:44:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.214+0000 7f702b7fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:44:34.197 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-03-29T10:44:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.215+0000 7f7049081640 10 librados: watch_flush enter 2024-03-29T10:44:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.215+0000 7f7049081640 10 librados: watch_flush exit 2024-03-29T10:44:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.215+0000 7f7049081640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T10:44:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.215+0000 7f7049081640 10 client.4113.objecter successfully canceled tick 2024-03-29T10:44:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.215+0000 7f7049081640 10 monclient: shutdown 2024-03-29T10:44:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.215+0000 7f7049081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.215+0000 7f7049081640 1 -- 172.21.15.98:0/719268542 >> v1:172.21.15.98:6789/0 conn(0x7f703000b020 legacy=0x7f70441e1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.215+0000 7f7049081640 1 -- 172.21.15.98:0/719268542 shutdown_connections 2024-03-29T10:44:34.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.215+0000 7f7049081640 1 -- 172.21.15.98:0/719268542 >> 172.21.15.98:0/719268542 conn(0x7f7044066b00 msgr2=0x7f70441456e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:34.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.215+0000 7f7049081640 1 -- 172.21.15.98:0/719268542 shutdown_connections 2024-03-29T10:44:34.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.215+0000 7f7049081640 1 -- 172.21.15.98:0/719268542 wait complete. 2024-03-29T10:44:34.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:34.215+0000 7f7049081640 1 librados: shutdown 2024-03-29T10:44:34.210 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:44:34.210 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:44:34.210 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:44:34.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:44:34.238 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:44:34.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:44:34.304 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:44:34.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:44:34.371 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-29T10:44:34.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:44:34.439 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:44:34.440 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:44:34.467 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:44:34.467 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:44:34.534 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:44:34.535 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:44:34.602 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-29T10:44:34.602 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:44:34.612 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:34.629+0000 7f546422c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:44:34.629 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:34.647+0000 7f3732c0f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:44:34.639 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 04bb1a53-61cb-479f-add3-3f9964d54955 0 2024-03-29T10:44:34.670 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:34.687+0000 7f546422c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:44:34.683 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:34.700+0000 7f3732c0f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:44:34.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.786+0000 7f3424dcb640 10 monclient: get_monmap_and_config 2024-03-29T10:44:34.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.786+0000 7f3424dcb640 10 monclient: build_initial_monmap 2024-03-29T10:44:34.770 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.787+0000 7f3424dcb640 10 monclient: monmap: 2024-03-29T10:44:34.770 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:34.770 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.770 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:44:34.787948+0000 2024-03-29T10:44:34.770 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:44:34.787948+0000 2024-03-29T10:44:34.771 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:34.771 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.771 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:34.771 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:34.771 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:34.771 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.787+0000 7f3424dcb640 1 Processor -- start 2024-03-29T10:44:34.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.787+0000 7f3424dcb640 1 -- start start 2024-03-29T10:44:34.771 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.787+0000 7f3424dcb640 10 monclient: init 2024-03-29T10:44:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f3424dcb640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f3424dcb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f3424dcb640 10 monclient(hunting): picked mon.noname-b con 0x7f342009ac40 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f3424dcb640 10 monclient(hunting): picked mon.noname-a con 0x7f342009c870 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f3424dcb640 10 monclient(hunting): picked mon.noname-c con 0x7f34200b4360 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f3424dcb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f342009dec0 con 0x7f342009c870 2024-03-29T10:44:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f3424dcb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f342009da30 con 0x7f34200b4360 2024-03-29T10:44:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f3424dcb640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34200b39c0 con 0x7f342009ac40 2024-03-29T10:44:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f3424dcb640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f3424dcb640 10 monclient(hunting): authenticate will time out at 885.082306s 2024-03-29T10:44:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341ed9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f34200b4360 0x7f342014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57130/0 (socket says 172.21.15.189:57130) 2024-03-29T10:44:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341ed9d640 1 -- 172.21.15.189:0/3432295541 learned_addr learned my addr 172.21.15.189:0/3432295541 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:34.781 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 1 -- 172.21.15.189:0/3432295541 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261655766 0 0) 0x7f342009da30 con 0x7f34200b4360 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient(hunting): my global_id is 4108 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 1 -- 172.21.15.189:0/3432295541 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f340c003520 con 0x7f34200b4360 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 1 -- 172.21.15.189:0/3432295541 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250638346 0 0) 0x7f3408001110 con 0x7f342009c870 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 1 -- 172.21.15.189:0/3432295541 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f342009da30 con 0x7f342009c870 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 1 -- 172.21.15.189:0/3432295541 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138155548 0 0) 0x7f34200b39c0 con 0x7f342009ac40 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.782 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient(hunting): my global_id is 4109 2024-03-29T10:44:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 1 -- 172.21.15.189:0/3432295541 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3408001110 con 0x7f342009ac40 2024-03-29T10:44:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 1 -- 172.21.15.189:0/3432295541 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3045250300 0 0) 0x7f340c003520 con 0x7f34200b4360 2024-03-29T10:44:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 1 -- 172.21.15.189:0/3432295541 >> v1:172.21.15.98:6790/0 conn(0x7f342009ac40 legacy=0x7f34200b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 1 -- 172.21.15.189:0/3432295541 >> v1:172.21.15.98:6789/0 conn(0x7f342009c870 legacy=0x7f342013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:44:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 1 monclient: found mon.noname-c 2024-03-29T10:44:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:44:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 1 -- 172.21.15.189:0/3432295541 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34200b5160 con 0x7f34200b4360 2024-03-29T10:44:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient: _finish_auth 0 2024-03-29T10:44:34.783 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 1 -- 172.21.15.189:0/3432295541 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3418001a10 con 0x7f34200b4360 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f3424dcb640 5 monclient: authenticate success, global_id 4108 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f3424dcb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.788+0000 7f341dd9b640 10 monclient: dump: 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.784 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f3424dcb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f341dd9b640 1 -- 172.21.15.189:0/3432295541 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3418001bf0 con 0x7f34200b4360 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f341dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f341dd9b640 1 -- 172.21.15.189:0/3432295541 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3418002000 con 0x7f34200b4360 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f3424dcb640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f341dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f3424dcb640 4 set_mon_vals no callback set 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f341dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f341dd9b640 10 monclient: dump: 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.785 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.786 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.786 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.786 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.786 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.786 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f3424dcb640 10 monclient: shutdown 2024-03-29T10:44:34.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f3424dcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:34.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f3424dcb640 1 -- 172.21.15.189:0/3432295541 >> v1:172.21.15.189:6789/0 conn(0x7f34200b4360 legacy=0x7f342014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f3424dcb640 1 -- 172.21.15.189:0/3432295541 shutdown_connections 2024-03-29T10:44:34.786 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f3424dcb640 1 -- 172.21.15.189:0/3432295541 >> 172.21.15.189:0/3432295541 conn(0x7f3420066b00 msgr2=0x7f3420066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:34.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f3424dcb640 1 -- 172.21.15.189:0/3432295541 shutdown_connections 2024-03-29T10:44:34.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.789+0000 7f3424dcb640 1 -- 172.21.15.189:0/3432295541 wait complete. 2024-03-29T10:44:34.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.790+0000 7f3424dcb640 10 monclient: build_initial_monmap 2024-03-29T10:44:34.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.790+0000 7f3424dcb640 10 monclient: monmap: 2024-03-29T10:44:34.787 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:34.787 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:34.787 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-03-29T10:44:34.787 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-03-29T10:44:34.788 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:34.788 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.788 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:34.788 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:34.788 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:34.788 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.790+0000 7f3424dcb640 1 librados: starting msgr at 2024-03-29T10:44:34.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.790+0000 7f3424dcb640 1 librados: starting objecter 2024-03-29T10:44:34.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.790+0000 7f3424dcb640 1 Processor -- start 2024-03-29T10:44:34.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.790+0000 7f3424dcb640 1 -- start start 2024-03-29T10:44:34.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.790+0000 7f3424dcb640 1 librados: setting wanted keys 2024-03-29T10:44:34.788 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.790+0000 7f3424dcb640 1 librados: calling monclient init 2024-03-29T10:44:34.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.790+0000 7f3424dcb640 10 monclient: init 2024-03-29T10:44:34.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3424dcb640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:34.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3424dcb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:34.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3424dcb640 10 monclient(hunting): picked mon.noname-b con 0x7f342009ac40 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:34.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3424dcb640 10 monclient(hunting): picked mon.noname-a con 0x7f34201d7cd0 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:34.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3424dcb640 10 monclient(hunting): picked mon.noname-c con 0x7f34200ad1e0 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:34.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3424dcb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34201d7130 con 0x7f34201d7cd0 2024-03-29T10:44:34.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3424dcb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3420139b10 con 0x7f342009ac40 2024-03-29T10:44:34.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3424dcb640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f342013ffa0 con 0x7f34200ad1e0 2024-03-29T10:44:34.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3424dcb640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:34.789 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3424dcb640 10 monclient(hunting): authenticate will time out at 885.085382s 2024-03-29T10:44:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f341e59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f342009ac40 0x7f34201d75c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57132/0 (socket says 172.21.15.189:57132) 2024-03-29T10:44:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f341e59c640 1 -- 172.21.15.189:0/1748529149 learned_addr learned my addr 172.21.15.189:0/1748529149 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 client.?.objecter ms_handle_connect 0x7f342009ac40 2024-03-29T10:44:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 client.?.objecter ms_handle_connect 0x7f34201d7cd0 2024-03-29T10:44:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 client.?.objecter ms_handle_connect 0x7f34200ad1e0 2024-03-29T10:44:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752492485 0 0) 0x7f3420139b10 con 0x7f342009ac40 2024-03-29T10:44:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.790 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:44:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3400004eb0 con 0x7f342009ac40 2024-03-29T10:44:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828820651 0 0) 0x7f342013ffa0 con 0x7f34200ad1e0 2024-03-29T10:44:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): my global_id is 4112 2024-03-29T10:44:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3420139b10 con 0x7f34200ad1e0 2024-03-29T10:44:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107535806 0 0) 0x7f34201d7130 con 0x7f34201d7cd0 2024-03-29T10:44:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.791 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f342013ffa0 con 0x7f34201d7cd0 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1474308315 0 0) 0x7f3400004eb0 con 0x7f342009ac40 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34201d7130 con 0x7f342009ac40 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3410001a10 con 0x7f342009ac40 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): dump: 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.792 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.791+0000 7f3416ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2377955862 0 0) 0x7f3420139b10 con 0x7f34200ad1e0 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3400004eb0 con 0x7f34200ad1e0 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3418001f00 con 0x7f34200ad1e0 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient(hunting): dump: 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.793 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2371695767 0 0) 0x7f342013ffa0 con 0x7f34201d7cd0 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3420139b10 con 0x7f34201d7cd0 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3408001860 con 0x7f34201d7cd0 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient(hunting): dump: 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.794 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1944288464 0 0) 0x7f34201d7130 con 0x7f342009ac40 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 >> v1:172.21.15.98:6790/0 conn(0x7f34200ad1e0 legacy=0x7f3420139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 >> v1:172.21.15.98:6789/0 conn(0x7f34201d7cd0 legacy=0x7f3420138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 monclient: found mon.b 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:34.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34200ae520 con 0x7f342009ac40 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3424dcb640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3424dcb640 10 monclient: _renew_subs 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3424dcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3424dcb640 1 -- 172.21.15.189:0/1748529149 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34200ae800 con 0x7f342009ac40 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3424dcb640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3424dcb640 10 monclient: _renew_subs 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3424dcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3424dcb640 1 -- 172.21.15.189:0/1748529149 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3420142b70 con 0x7f342009ac40 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3424dcb640 1 librados: init done 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3410001cf0 con 0x7f342009ac40 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:34.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3410003a60 con 0x7f342009ac40 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 monclient: dump: 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.797 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f341d59a640 4 set_mon_vals no callback set 2024-03-29T10:44:34.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3410004010 con 0x7f342009ac40 2024-03-29T10:44:34.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 976802295 0 0) 0x7f34100066d0 con 0x7f342009ac40 2024-03-29T10:44:34.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 10 client.4111.objecter ms_dispatch 0x7f3420000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:44:34.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:44:34.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:44:34.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.792+0000 7f3416ffd640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T10:44:34.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.793+0000 7f3424dcb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.793+0000 7f3424dcb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.793+0000 7f3424dcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:34.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.793+0000 7f3424dcb640 1 -- 172.21.15.189:0/1748529149 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34201d80b0 con 0x7f342009ac40 2024-03-29T10:44:34.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.795+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f34100069b0 con 0x7f342009ac40 2024-03-29T10:44:34.798 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.795+0000 7f3416ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.799 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.795+0000 7f3416ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:34.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.867+0000 7f3424dcb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "04bb1a53-61cb-479f-add3-3f9964d54955", "id": 0}] 2024-03-29T10:44:34.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.867+0000 7f3424dcb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "04bb1a53-61cb-479f-add3-3f9964d54955", "id": 0}] 2024-03-29T10:44:34.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.867+0000 7f3424dcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:34.851 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.867+0000 7f3424dcb640 1 -- 172.21.15.189:0/1748529149 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "04bb1a53-61cb-479f-add3-3f9964d54955", "id": 0} v 0) v1 -- 0x7f342004f730 con 0x7f342009ac40 2024-03-29T10:44:34.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.873+0000 7f3416ffd640 1 -- 172.21.15.189:0/1748529149 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "04bb1a53-61cb-479f-add3-3f9964d54955", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3983626406 0 732707515) 0x7f3410006b50 con 0x7f342009ac40 2024-03-29T10:44:34.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.873+0000 7f3416ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "04bb1a53-61cb-479f-add3-3f9964d54955", "id": 0}] 2024-03-29T10:44:34.856 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.873+0000 7f3416ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:34.856 INFO:teuthology.orchestra.run.smithi189.stdout:0 2024-03-29T10:44:34.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.874+0000 7f3424dcb640 10 librados: watch_flush enter 2024-03-29T10:44:34.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.874+0000 7f3424dcb640 10 librados: watch_flush exit 2024-03-29T10:44:34.857 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.874+0000 7f3424dcb640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T10:44:34.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.874+0000 7f3424dcb640 10 client.4111.objecter successfully canceled tick 2024-03-29T10:44:34.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.874+0000 7f3424dcb640 10 monclient: shutdown 2024-03-29T10:44:34.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.874+0000 7f3424dcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:34.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.874+0000 7f3424dcb640 1 -- 172.21.15.189:0/1748529149 >> v1:172.21.15.189:6789/0 conn(0x7f342009ac40 legacy=0x7f34201d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.874+0000 7f3424dcb640 1 -- 172.21.15.189:0/1748529149 shutdown_connections 2024-03-29T10:44:34.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.874+0000 7f3424dcb640 1 -- 172.21.15.189:0/1748529149 >> 172.21.15.189:0/1748529149 conn(0x7f3420066b00 msgr2=0x7f34200b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:34.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.874+0000 7f3424dcb640 1 -- 172.21.15.189:0/1748529149 shutdown_connections 2024-03-29T10:44:34.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.874+0000 7f3424dcb640 1 -- 172.21.15.189:0/1748529149 wait complete. 2024-03-29T10:44:34.858 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.874+0000 7f3424dcb640 1 librados: shutdown 2024-03-29T10:44:34.867 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new bd57627e-f989-44a1-ba96-4fb20f6191ed 1 2024-03-29T10:44:34.966 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.983+0000 7feef274b640 10 monclient: get_monmap_and_config 2024-03-29T10:44:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.983+0000 7feef274b640 10 monclient: build_initial_monmap 2024-03-29T10:44:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.983+0000 7feef274b640 10 monclient: monmap: 2024-03-29T10:44:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:44:34.984930+0000 2024-03-29T10:44:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:44:34.984930+0000 2024-03-29T10:44:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:34.967 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:34.967 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.984+0000 7feef274b640 1 Processor -- start 2024-03-29T10:44:34.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.984+0000 7feef274b640 1 -- start start 2024-03-29T10:44:34.976 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.984+0000 7feef274b640 10 monclient: init 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feef274b640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feef274b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feef274b640 10 monclient(hunting): picked mon.noname-a con 0x7feeec0b1780 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feef274b640 10 monclient(hunting): picked mon.noname-c con 0x7feeec13d7e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feef274b640 10 monclient(hunting): picked mon.noname-b con 0x7feeec143be0 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feef274b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeec070b30 con 0x7feeec0b1780 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feef274b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeec070da0 con 0x7feeec13d7e0 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feef274b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeec09adb0 con 0x7feeec143be0 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feef274b640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feef274b640 10 monclient(hunting): authenticate will time out at 885.279360s 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feeeb7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7feeec13d7e0 0x7feeec1434d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57144/0 (socket says 172.21.15.189:57144) 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feeeb7fe640 1 -- 172.21.15.189:0/3094221706 learned_addr learned my addr 172.21.15.189:0/3094221706 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feeeaffd640 1 -- 172.21.15.189:0/3094221706 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560439567 0 0) 0x7feeec070da0 con 0x7feeec13d7e0 2024-03-29T10:44:34.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feeeaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feeeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feeeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feeeaffd640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:44:34.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feeeaffd640 1 -- 172.21.15.189:0/3094221706 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed4003520 con 0x7feeec13d7e0 2024-03-29T10:44:34.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.985+0000 7feeeaffd640 1 -- 172.21.15.189:0/3094221706 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1025931776 0 0) 0x7feed4003520 con 0x7feeec13d7e0 2024-03-29T10:44:34.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 1 -- 172.21.15.189:0/3094221706 >> v1:172.21.15.98:6790/0 conn(0x7feeec143be0 legacy=0x7feeec14b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 1 -- 172.21.15.189:0/3094221706 >> v1:172.21.15.98:6789/0 conn(0x7feeec0b1780 legacy=0x7feeec13d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:34.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 1 monclient: found mon.noname-c 2024-03-29T10:44:34.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:34.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:44:34.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 1 -- 172.21.15.189:0/3094221706 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec144070 con 0x7feeec13d7e0 2024-03-29T10:44:34.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 1 -- 172.21.15.189:0/3094221706 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7feedc001690 con 0x7feeec13d7e0 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feef274b640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feef274b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 10 monclient: dump: 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.979 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feef274b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 1 -- 172.21.15.189:0/3094221706 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feedc001870 con 0x7feeec13d7e0 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 1 -- 172.21.15.189:0/3094221706 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7feedc001cc0 con 0x7feeec13d7e0 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feef274b640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feef274b640 4 set_mon_vals no callback set 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feeeaffd640 10 monclient: dump: 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:34.980 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feef274b640 10 monclient: shutdown 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feef274b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feef274b640 1 -- 172.21.15.189:0/3094221706 >> v1:172.21.15.189:6789/0 conn(0x7feeec13d7e0 legacy=0x7feeec1434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feef274b640 1 -- 172.21.15.189:0/3094221706 shutdown_connections 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feef274b640 1 -- 172.21.15.189:0/3094221706 >> 172.21.15.189:0/3094221706 conn(0x7feeec0b58f0 msgr2=0x7feeec0b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feef274b640 1 -- 172.21.15.189:0/3094221706 shutdown_connections 2024-03-29T10:44:34.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.986+0000 7feef274b640 1 -- 172.21.15.189:0/3094221706 wait complete. 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.987+0000 7feef274b640 10 monclient: build_initial_monmap 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.987+0000 7feef274b640 10 monclient: monmap: 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.987+0000 7feef274b640 1 librados: starting msgr at 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.987+0000 7feef274b640 1 librados: starting objecter 2024-03-29T10:44:34.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.987+0000 7feef274b640 1 Processor -- start 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.987+0000 7feef274b640 1 -- start start 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.987+0000 7feef274b640 1 librados: setting wanted keys 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.987+0000 7feef274b640 1 librados: calling monclient init 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.987+0000 7feef274b640 10 monclient: init 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feef274b640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feef274b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feef274b640 10 monclient(hunting): picked mon.noname-a con 0x7feeec0b1780 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feef274b640 10 monclient(hunting): picked mon.noname-b con 0x7feeec13d7e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feef274b640 10 monclient(hunting): picked mon.noname-c con 0x7feeec143be0 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feef274b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeec1d8210 con 0x7feeec0b1780 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feef274b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeec13a540 con 0x7feeec13d7e0 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feef274b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeec138c60 con 0x7feeec143be0 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feef274b640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:34.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feef274b640 10 monclient(hunting): authenticate will time out at 885.282450s 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feeeb7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7feeec13d7e0 0x7feeec1d8730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57150/0 (socket says 172.21.15.189:57150) 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feeeb7fe640 1 -- 172.21.15.189:0/3374477310 learned_addr learned my addr 172.21.15.189:0/3374477310 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 client.?.objecter ms_handle_connect 0x7feeec13d7e0 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 client.?.objecter ms_handle_connect 0x7feeec0b1780 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 client.?.objecter ms_handle_connect 0x7feeec143be0 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028867587 0 0) 0x7feeec13a540 con 0x7feeec13d7e0 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:44:34.984 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed8004eb0 con 0x7feeec13d7e0 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414211217 0 0) 0x7feeec1d8210 con 0x7feeec0b1780 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeec13a540 con 0x7feeec0b1780 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113868274 0 0) 0x7feeec138c60 con 0x7feeec143be0 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.988+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeec1d8210 con 0x7feeec143be0 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3932492797 0 0) 0x7feed8004eb0 con 0x7feeec13d7e0 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feeec138c60 con 0x7feeec13d7e0 2024-03-29T10:44:34.985 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7feedc001bc0 con 0x7feeec13d7e0 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.986 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1859042878 0 0) 0x7feeec13a540 con 0x7feeec0b1780 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feed8004eb0 con 0x7feeec0b1780 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1709205139 0 0) 0x7feeec1d8210 con 0x7feeec143be0 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feeec13a540 con 0x7feeec143be0 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7feee00017f0 con 0x7feeec0b1780 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.987 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7feee4001940 con 0x7feeec143be0 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.988 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2338755826 0 0) 0x7feeec138c60 con 0x7feeec13d7e0 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 >> v1:172.21.15.98:6790/0 conn(0x7feeec143be0 legacy=0x7feeec1df710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 >> v1:172.21.15.98:6789/0 conn(0x7feeec0b1780 legacy=0x7feeec139240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 monclient: found mon.b 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec138df0 con 0x7feeec13d7e0 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:44:34.989 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feef274b640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feef274b640 10 monclient: _renew_subs 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feef274b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feef274b640 1 -- 172.21.15.189:0/3374477310 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeec139950 con 0x7feeec13d7e0 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feef274b640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feef274b640 10 monclient: _renew_subs 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feef274b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feef274b640 1 -- 172.21.15.189:0/3374477310 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feeec141c40 con 0x7feeec13d7e0 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feef274b640 1 librados: init done 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feedc004bf0 con 0x7feeec13d7e0 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7feedc004f00 con 0x7feeec13d7e0 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 monclient: dump: 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feeea7fc640 4 set_mon_vals no callback set 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7feedc0053f0 con 0x7feeec13d7e0 2024-03-29T10:44:34.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3940645823 0 0) 0x7feedc006960 con 0x7feeec13d7e0 2024-03-29T10:44:34.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 10 client.4117.objecter ms_dispatch 0x7feeec000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:44:34.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:44:34.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:44:34.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.989+0000 7feee8ff9640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T10:44:34.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.990+0000 7feef274b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.990+0000 7feef274b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.990+0000 7feef274b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:34.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.990+0000 7feef274b640 1 -- 172.21.15.189:0/3374477310 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeec067c00 con 0x7feeec13d7e0 2024-03-29T10:44:34.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.992+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7feedc006ba0 con 0x7feeec13d7e0 2024-03-29T10:44:34.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.992+0000 7feee8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.992 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:34.992+0000 7feee8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:35.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.064+0000 7feef274b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bd57627e-f989-44a1-ba96-4fb20f6191ed", "id": 1}] 2024-03-29T10:44:35.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.064+0000 7feef274b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bd57627e-f989-44a1-ba96-4fb20f6191ed", "id": 1}] 2024-03-29T10:44:35.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.064+0000 7feef274b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.047 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.064+0000 7feef274b640 1 -- 172.21.15.189:0/3374477310 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bd57627e-f989-44a1-ba96-4fb20f6191ed", "id": 1} v 0) v1 -- 0x7feeec04f730 con 0x7feeec13d7e0 2024-03-29T10:44:35.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.072+0000 7feee8ff9640 1 -- 172.21.15.189:0/3374477310 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bd57627e-f989-44a1-ba96-4fb20f6191ed", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2151623472 0 940483276) 0x7feedc0059b0 con 0x7feeec13d7e0 2024-03-29T10:44:35.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.072+0000 7feee8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bd57627e-f989-44a1-ba96-4fb20f6191ed", "id": 1}] 2024-03-29T10:44:35.055 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.072+0000 7feee8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:35.055 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-03-29T10:44:35.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.073+0000 7feef274b640 10 librados: watch_flush enter 2024-03-29T10:44:35.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.073+0000 7feef274b640 10 librados: watch_flush exit 2024-03-29T10:44:35.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.073+0000 7feef274b640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T10:44:35.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.073+0000 7feef274b640 10 client.4117.objecter successfully canceled tick 2024-03-29T10:44:35.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.073+0000 7feef274b640 10 monclient: shutdown 2024-03-29T10:44:35.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.073+0000 7feef274b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:35.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.073+0000 7feef274b640 1 -- 172.21.15.189:0/3374477310 >> v1:172.21.15.189:6789/0 conn(0x7feeec13d7e0 legacy=0x7feeec1d8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.073+0000 7feef274b640 1 -- 172.21.15.189:0/3374477310 shutdown_connections 2024-03-29T10:44:35.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.073+0000 7feef274b640 1 -- 172.21.15.189:0/3374477310 >> 172.21.15.189:0/3374477310 conn(0x7feeec0b58f0 msgr2=0x7feeec069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:35.057 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.073+0000 7feef274b640 1 -- 172.21.15.189:0/3374477310 shutdown_connections 2024-03-29T10:44:35.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.073+0000 7feef274b640 1 -- 172.21.15.189:0/3374477310 wait complete. 2024-03-29T10:44:35.058 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.073+0000 7feef274b640 1 librados: shutdown 2024-03-29T10:44:35.066 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 89e48e12-41c1-408c-b994-f567d9c3c139 2 2024-03-29T10:44:35.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.185+0000 7fe2a1e5c640 10 monclient: get_monmap_and_config 2024-03-29T10:44:35.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.185+0000 7fe2a1e5c640 10 monclient: build_initial_monmap 2024-03-29T10:44:35.168 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.185+0000 7fe2a1e5c640 10 monclient: monmap: 2024-03-29T10:44:35.168 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:35.168 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.168 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:44:35.186320+0000 2024-03-29T10:44:35.168 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:44:35.186320+0000 2024-03-29T10:44:35.168 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:35.169 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.169 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:35.169 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:35.169 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:35.169 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.178 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe2a1e5c640 1 Processor -- start 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe2a1e5c640 1 -- start start 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe2a1e5c640 10 monclient: init 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe2a1e5c640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe2a1e5c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe2a1e5c640 10 monclient(hunting): picked mon.noname-b con 0x7fe29c09ac40 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe2a1e5c640 10 monclient(hunting): picked mon.noname-c con 0x7fe29c09c870 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe2a1e5c640 10 monclient(hunting): picked mon.noname-a con 0x7fe29c0b4360 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe2a1e5c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c09dec0 con 0x7fe29c0b4360 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe2a1e5c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c09da30 con 0x7fe29c09c870 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe2a1e5c640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c0b39c0 con 0x7fe29c09ac40 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe2a1e5c640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe2a1e5c640 10 monclient(hunting): authenticate will time out at 885.480781s 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe29affd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe29c09c870 0x7fe29c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57166/0 (socket says 172.21.15.189:57166) 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.186+0000 7fe29affd640 1 -- 172.21.15.189:0/2713008045 learned_addr learned my addr 172.21.15.189:0/2713008045 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:35.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 -- 172.21.15.189:0/2713008045 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299139550 0 0) 0x7fe29c09da30 con 0x7fe29c09c870 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 -- 172.21.15.189:0/2713008045 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe284003520 con 0x7fe29c09c870 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 -- 172.21.15.189:0/2713008045 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822285878 0 0) 0x7fe294001490 con 0x7fe29c0b4360 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 -- 172.21.15.189:0/2713008045 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe29c09da30 con 0x7fe29c0b4360 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 -- 172.21.15.189:0/2713008045 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170025147 0 0) 0x7fe29c0b39c0 con 0x7fe29c09ac40 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.180 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 -- 172.21.15.189:0/2713008045 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe294001490 con 0x7fe29c09ac40 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 -- 172.21.15.189:0/2713008045 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4168691317 0 0) 0x7fe284003520 con 0x7fe29c09c870 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 -- 172.21.15.189:0/2713008045 >> v1:172.21.15.98:6790/0 conn(0x7fe29c09ac40 legacy=0x7fe29c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 -- 172.21.15.189:0/2713008045 >> v1:172.21.15.98:6789/0 conn(0x7fe29c0b4360 legacy=0x7fe29c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 -- 172.21.15.189:0/2713008045 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c0b5160 con 0x7fe29c09c870 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 -- 172.21.15.189:0/2713008045 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe290001690 con 0x7fe29c09c870 2024-03-29T10:44:35.181 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe2a1e5c640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe2a1e5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient: dump: 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.182 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe2a1e5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 -- 172.21.15.189:0/2713008045 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe290001870 con 0x7fe29c09c870 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe2a1e5c640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 1 -- 172.21.15.189:0/2713008045 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe290001cc0 con 0x7fe29c09c870 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe2a1e5c640 4 set_mon_vals no callback set 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.187+0000 7fe29a7fc640 10 monclient: dump: 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.183 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.188+0000 7fe2a1e5c640 10 monclient: shutdown 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.188+0000 7fe2a1e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.188+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/2713008045 >> v1:172.21.15.189:6789/0 conn(0x7fe29c09c870 legacy=0x7fe29c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.188+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/2713008045 shutdown_connections 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.188+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/2713008045 >> 172.21.15.189:0/2713008045 conn(0x7fe29c066b00 msgr2=0x7fe29c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.188+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/2713008045 shutdown_connections 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.188+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/2713008045 wait complete. 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.188+0000 7fe2a1e5c640 10 monclient: build_initial_monmap 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.188+0000 7fe2a1e5c640 10 monclient: monmap: 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:35.184 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.189+0000 7fe2a1e5c640 1 librados: starting msgr at 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.189+0000 7fe2a1e5c640 1 librados: starting objecter 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.189+0000 7fe2a1e5c640 1 Processor -- start 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.189+0000 7fe2a1e5c640 1 -- start start 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.189+0000 7fe2a1e5c640 1 librados: setting wanted keys 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.189+0000 7fe2a1e5c640 1 librados: calling monclient init 2024-03-29T10:44:35.185 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.189+0000 7fe2a1e5c640 10 monclient: init 2024-03-29T10:44:35.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.189+0000 7fe2a1e5c640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:35.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.189+0000 7fe2a1e5c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:35.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.189+0000 7fe2a1e5c640 10 monclient(hunting): picked mon.noname-b con 0x7fe29c09ac40 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:35.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.189+0000 7fe2a1e5c640 10 monclient(hunting): picked mon.noname-a con 0x7fe29c145540 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:35.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.189+0000 7fe2a1e5c640 10 monclient(hunting): picked mon.noname-c con 0x7fe29c1434a0 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:35.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c139420 con 0x7fe29c145540 2024-03-29T10:44:35.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c1395b0 con 0x7fe29c09ac40 2024-03-29T10:44:35.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c139740 con 0x7fe29c1434a0 2024-03-29T10:44:35.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:35.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 10 monclient(hunting): authenticate will time out at 885.483991s 2024-03-29T10:44:35.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe29b7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe29c09ac40 0x7fe29c138d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57172/0 (socket says 172.21.15.189:57172) 2024-03-29T10:44:35.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe29b7fe640 1 -- 172.21.15.189:0/1898306199 learned_addr learned my addr 172.21.15.189:0/1898306199 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:35.186 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 client.?.objecter ms_handle_connect 0x7fe29c09ac40 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 client.?.objecter ms_handle_connect 0x7fe29c145540 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 client.?.objecter ms_handle_connect 0x7fe29c1434a0 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438335366 0 0) 0x7fe29c139420 con 0x7fe29c145540 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe288004eb0 con 0x7fe29c145540 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662861463 0 0) 0x7fe29c1395b0 con 0x7fe29c09ac40 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe29c139420 con 0x7fe29c09ac40 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106757981 0 0) 0x7fe29c139740 con 0x7fe29c1434a0 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe29c1395b0 con 0x7fe29c1434a0 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1658459963 0 0) 0x7fe288004eb0 con 0x7fe29c145540 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe29c139740 con 0x7fe29c145540 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe290001b80 con 0x7fe29c145540 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): dump: 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.188 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3141549982 0 0) 0x7fe29c139420 con 0x7fe29c09ac40 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe288004eb0 con 0x7fe29c09ac40 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 495478428 0 0) 0x7fe29c1395b0 con 0x7fe29c1434a0 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe29c139420 con 0x7fe29c1434a0 2024-03-29T10:44:35.189 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe28c0018b0 con 0x7fe29c09ac40 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): dump: 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.190 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe294001bc0 con 0x7fe29c1434a0 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient(hunting): dump: 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 584666092 0 0) 0x7fe29c139740 con 0x7fe29c145540 2024-03-29T10:44:35.191 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 >> v1:172.21.15.98:6790/0 conn(0x7fe29c1434a0 legacy=0x7fe29c1d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 >> v1:172.21.15.189:6789/0 conn(0x7fe29c09ac40 legacy=0x7fe29c138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 monclient: found mon.a 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c0acd00 con 0x7fe29c145540 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient: _finish_auth 0 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 5 monclient: authenticate success, global_id 4131 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 10 monclient: _renew_subs 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/1898306199 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29c0acfb0 con 0x7fe29c145540 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:35.192 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 10 monclient: _renew_subs 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/1898306199 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29c1e37c0 con 0x7fe29c145540 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe2a1e5c640 1 librados: init done 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe290004bf0 con 0x7fe29c145540 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe290004f00 con 0x7fe29c145540 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe299ffb640 4 set_mon_vals no callback set 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 monclient: dump: 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.193 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe2900053f0 con 0x7fe29c145540 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2545967924 0 0) 0x7fe290006880 con 0x7fe29c145540 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 10 client.4131.objecter ms_dispatch 0x7fe29c000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.190+0000 7fe283fff640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.191+0000 7fe283fff640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.191+0000 7fe2a1e5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.191+0000 7fe2a1e5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:35.194 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.191+0000 7fe2a1e5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:35.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.191+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/1898306199 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe27c005080 con 0x7fe29c145540 2024-03-29T10:44:35.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.193+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe290006510 con 0x7fe29c145540 2024-03-29T10:44:35.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.193+0000 7fe283fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:35.195 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.193+0000 7fe283fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:35.230 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:35.248+0000 7f3732c0f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:44:35.234 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:35.251+0000 7f546422c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:44:35.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.266+0000 7fe2a1e5c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "89e48e12-41c1-408c-b994-f567d9c3c139", "id": 2}] 2024-03-29T10:44:35.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.266+0000 7fe2a1e5c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "89e48e12-41c1-408c-b994-f567d9c3c139", "id": 2}] 2024-03-29T10:44:35.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.266+0000 7fe2a1e5c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:35.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.266+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/1898306199 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd new", "uuid": "89e48e12-41c1-408c-b994-f567d9c3c139", "id": 2} v 0) v1 -- 0x7fe27c0058b0 con 0x7fe29c145540 2024-03-29T10:44:35.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.268+0000 7fe283fff640 1 -- 172.21.15.189:0/1898306199 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "89e48e12-41c1-408c-b994-f567d9c3c139", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3915119547 0 216599125) 0x7fe290006a40 con 0x7fe29c145540 2024-03-29T10:44:35.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.268+0000 7fe283fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "89e48e12-41c1-408c-b994-f567d9c3c139", "id": 2}] 2024-03-29T10:44:35.250 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.268+0000 7fe283fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:35.251 INFO:teuthology.orchestra.run.smithi189.stdout:2 2024-03-29T10:44:35.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.268+0000 7fe2a1e5c640 10 librados: watch_flush enter 2024-03-29T10:44:35.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.268+0000 7fe2a1e5c640 10 librados: watch_flush exit 2024-03-29T10:44:35.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.268+0000 7fe2a1e5c640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-29T10:44:35.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.268+0000 7fe2a1e5c640 10 client.4131.objecter successfully canceled tick 2024-03-29T10:44:35.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.268+0000 7fe2a1e5c640 10 monclient: shutdown 2024-03-29T10:44:35.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.268+0000 7fe2a1e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:35.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.268+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/1898306199 >> v1:172.21.15.98:6789/0 conn(0x7fe29c145540 legacy=0x7fe29c1d6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.269+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/1898306199 shutdown_connections 2024-03-29T10:44:35.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.269+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/1898306199 >> 172.21.15.189:0/1898306199 conn(0x7fe29c066b00 msgr2=0x7fe29c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:35.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.269+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/1898306199 shutdown_connections 2024-03-29T10:44:35.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.269+0000 7fe2a1e5c640 1 -- 172.21.15.189:0/1898306199 wait complete. 2024-03-29T10:44:35.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.269+0000 7fe2a1e5c640 1 librados: shutdown 2024-03-29T10:44:35.262 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30 3 2024-03-29T10:44:35.284 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:35.302+0000 7f3732c0f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:44:35.292 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:35.309+0000 7f546422c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:44:35.329 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:35.347+0000 7f3732c0f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:44:35.339 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:35.357+0000 7f546422c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:44:35.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.379+0000 7f76e6924640 10 monclient: get_monmap_and_config 2024-03-29T10:44:35.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.379+0000 7f76e6924640 10 monclient: build_initial_monmap 2024-03-29T10:44:35.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.379+0000 7f76e6924640 10 monclient: monmap: 2024-03-29T10:44:35.362 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:35.362 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.363 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:44:35.380787+0000 2024-03-29T10:44:35.363 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:44:35.380787+0000 2024-03-29T10:44:35.363 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:35.363 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.363 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:35.363 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:35.363 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:35.363 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.380+0000 7f76e6924640 1 Processor -- start 2024-03-29T10:44:35.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.380+0000 7f76e6924640 1 -- start start 2024-03-29T10:44:35.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.380+0000 7f76e6924640 10 monclient: init 2024-03-29T10:44:35.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76e6924640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76e6924640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76e6924640 10 monclient(hunting): picked mon.noname-b con 0x7f76e009ac40 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76e6924640 10 monclient(hunting): picked mon.noname-a con 0x7f76e009c870 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76e6924640 10 monclient(hunting): picked mon.noname-c con 0x7f76e00b4360 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76e6924640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e009dec0 con 0x7f76e009c870 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76e6924640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e009da30 con 0x7f76e00b4360 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76e6924640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e00b39c0 con 0x7f76e009ac40 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76e6924640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76e6924640 10 monclient(hunting): authenticate will time out at 885.675286s 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76e4ec1640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f76e00b4360 0x7f76e014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57178/0 (socket says 172.21.15.189:57178) 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76e4ec1640 1 -- 172.21.15.189:0/491997934 learned_addr learned my addr 172.21.15.189:0/491997934 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 1 -- 172.21.15.189:0/491997934 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491336033 0 0) 0x7f76e009da30 con 0x7f76e00b4360 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 1 -- 172.21.15.189:0/491997934 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c8003520 con 0x7f76e00b4360 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 1 -- 172.21.15.189:0/491997934 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025467272 0 0) 0x7f76d0001110 con 0x7f76e009c870 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 1 -- 172.21.15.189:0/491997934 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e009da30 con 0x7f76e009c870 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 1 -- 172.21.15.189:0/491997934 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023070911 0 0) 0x7f76e00b39c0 con 0x7f76e009ac40 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 1 -- 172.21.15.189:0/491997934 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d0001110 con 0x7f76e009ac40 2024-03-29T10:44:35.376 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 1 -- 172.21.15.189:0/491997934 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1382703135 0 0) 0x7f76c8003520 con 0x7f76e00b4360 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 1 -- 172.21.15.189:0/491997934 >> v1:172.21.15.98:6790/0 conn(0x7f76e009ac40 legacy=0x7f76e00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 1 -- 172.21.15.189:0/491997934 >> v1:172.21.15.98:6789/0 conn(0x7f76e009c870 legacy=0x7f76e013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 1 monclient: found mon.noname-c 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 1 -- 172.21.15.189:0/491997934 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e00b5160 con 0x7f76e00b4360 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76deffd640 1 -- 172.21.15.189:0/491997934 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f76d8001a10 con 0x7f76e00b4360 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.381+0000 7f76e6924640 5 monclient: authenticate success, global_id 4126 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76e6924640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.377 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76deffd640 10 monclient: dump: 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76e6924640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76deffd640 1 -- 172.21.15.189:0/491997934 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76d8001bf0 con 0x7f76e00b4360 2024-03-29T10:44:35.378 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:35.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76e6924640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:35.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76deffd640 1 -- 172.21.15.189:0/491997934 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f76d8002040 con 0x7f76e00b4360 2024-03-29T10:44:35.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76e6924640 4 set_mon_vals no callback set 2024-03-29T10:44:35.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:35.379 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76deffd640 10 monclient: dump: 2024-03-29T10:44:35.379 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.379 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.379 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76e6924640 10 monclient: shutdown 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76e6924640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76e6924640 1 -- 172.21.15.189:0/491997934 >> v1:172.21.15.189:6789/0 conn(0x7f76e00b4360 legacy=0x7f76e014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76e6924640 1 -- 172.21.15.189:0/491997934 shutdown_connections 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76e6924640 1 -- 172.21.15.189:0/491997934 >> 172.21.15.189:0/491997934 conn(0x7f76e0066b00 msgr2=0x7f76e0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76e6924640 1 -- 172.21.15.189:0/491997934 shutdown_connections 2024-03-29T10:44:35.380 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.382+0000 7f76e6924640 1 -- 172.21.15.189:0/491997934 wait complete. 2024-03-29T10:44:35.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.383+0000 7f76e6924640 10 monclient: build_initial_monmap 2024-03-29T10:44:35.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.383+0000 7f76e6924640 10 monclient: monmap: 2024-03-29T10:44:35.381 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:35.381 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:35.381 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-03-29T10:44:35.381 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-03-29T10:44:35.381 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:35.381 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.381 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:35.381 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:35.381 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:35.382 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.383+0000 7f76e6924640 1 librados: starting msgr at 2024-03-29T10:44:35.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.383+0000 7f76e6924640 1 librados: starting objecter 2024-03-29T10:44:35.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.383+0000 7f76e6924640 1 Processor -- start 2024-03-29T10:44:35.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.383+0000 7f76e6924640 1 -- start start 2024-03-29T10:44:35.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.383+0000 7f76e6924640 1 librados: setting wanted keys 2024-03-29T10:44:35.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.383+0000 7f76e6924640 1 librados: calling monclient init 2024-03-29T10:44:35.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.383+0000 7f76e6924640 10 monclient: init 2024-03-29T10:44:35.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76e6924640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:35.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76e6924640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:35.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76e6924640 10 monclient(hunting): picked mon.noname-a con 0x7f76e009ac40 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:35.382 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76e6924640 10 monclient(hunting): picked mon.noname-c con 0x7f76e0142e50 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:35.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76e6924640 10 monclient(hunting): picked mon.noname-b con 0x7f76e01db110 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:35.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76e6924640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e00ae2f0 con 0x7f76e009ac40 2024-03-29T10:44:35.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76e6924640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e00b1aa0 con 0x7f76e01db110 2024-03-29T10:44:35.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76e6924640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e013ffa0 con 0x7f76e0142e50 2024-03-29T10:44:35.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76e6924640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:35.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76e6924640 10 monclient(hunting): authenticate will time out at 885.678472s 2024-03-29T10:44:35.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76e4ec1640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f76e01db110 0x7f76e0138d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57180/0 (socket says 172.21.15.189:57180) 2024-03-29T10:44:35.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76e4ec1640 1 -- 172.21.15.189:0/3996222656 learned_addr learned my addr 172.21.15.189:0/3996222656 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:35.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 client.?.objecter ms_handle_connect 0x7f76e01db110 2024-03-29T10:44:35.383 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 client.?.objecter ms_handle_connect 0x7f76e009ac40 2024-03-29T10:44:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 client.?.objecter ms_handle_connect 0x7f76e0142e50 2024-03-29T10:44:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778615291 0 0) 0x7f76e00b1aa0 con 0x7f76e01db110 2024-03-29T10:44:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:44:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76cc004eb0 con 0x7f76e01db110 2024-03-29T10:44:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620842598 0 0) 0x7f76e00ae2f0 con 0x7f76e009ac40 2024-03-29T10:44:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.384 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:44:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e00b1aa0 con 0x7f76e009ac40 2024-03-29T10:44:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175693874 0 0) 0x7f76e013ffa0 con 0x7f76e0142e50 2024-03-29T10:44:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:44:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.384+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e00ae2f0 con 0x7f76e0142e50 2024-03-29T10:44:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 906712364 0 0) 0x7f76cc004eb0 con 0x7f76e01db110 2024-03-29T10:44:35.385 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76e013ffa0 con 0x7f76e01db110 2024-03-29T10:44:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f76d8001f40 con 0x7f76e01db110 2024-03-29T10:44:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient(hunting): dump: 2024-03-29T10:44:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.386 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.387 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2950405134 0 0) 0x7f76e00b1aa0 con 0x7f76e009ac40 2024-03-29T10:44:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76cc004eb0 con 0x7f76e009ac40 2024-03-29T10:44:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 751320860 0 0) 0x7f76e00ae2f0 con 0x7f76e0142e50 2024-03-29T10:44:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76e00b1aa0 con 0x7f76e0142e50 2024-03-29T10:44:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f76d4001870 con 0x7f76e009ac40 2024-03-29T10:44:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.387 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:35.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient(hunting): dump: 2024-03-29T10:44:35.388 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.388 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.388 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.388 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.388 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.388 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.388 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.388 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.389 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f76d0001a40 con 0x7f76e0142e50 2024-03-29T10:44:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient(hunting): dump: 2024-03-29T10:44:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.389 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.390 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.390 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.390 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.390 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.390 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.390 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2599087601 0 0) 0x7f76e013ffa0 con 0x7f76e01db110 2024-03-29T10:44:35.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 >> v1:172.21.15.98:6790/0 conn(0x7f76e0142e50 legacy=0x7f76e01daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.390 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 >> v1:172.21.15.98:6789/0 conn(0x7f76e009ac40 legacy=0x7f76e0142580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient: _finish_hunting 0 2024-03-29T10:44:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 monclient: found mon.b 2024-03-29T10:44:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e0140130 con 0x7f76e01db110 2024-03-29T10:44:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient: _finish_auth 0 2024-03-29T10:44:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient: _check_auth_tickets 2024-03-29T10:44:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76e6924640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:44:35.391 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76e6924640 10 monclient: _renew_subs 2024-03-29T10:44:35.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76e6924640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76e6924640 1 -- 172.21.15.189:0/3996222656 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e0139420 con 0x7f76e01db110 2024-03-29T10:44:35.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76e6924640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:35.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76e6924640 10 monclient: _renew_subs 2024-03-29T10:44:35.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76e6924640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76e6924640 1 -- 172.21.15.189:0/3996222656 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e01e37f0 con 0x7f76e01db110 2024-03-29T10:44:35.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76e6924640 1 librados: init done 2024-03-29T10:44:35.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76d8004e10 con 0x7f76e01db110 2024-03-29T10:44:35.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:35.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f76d8005120 con 0x7f76e01db110 2024-03-29T10:44:35.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:35.392 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76de7fc640 4 set_mon_vals no callback set 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 monclient: dump: 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f76d8005610 con 0x7f76e01db110 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3601461697 0 0) 0x7f76d8006a70 con 0x7f76e01db110 2024-03-29T10:44:35.393 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 10 client.4129.objecter ms_dispatch 0x7f76e0000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:44:35.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:44:35.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:44:35.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.385+0000 7f76dcff9640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T10:44:35.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.386+0000 7f76e6924640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:35.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.386+0000 7f76e6924640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:35.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.386+0000 7f76e6924640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.386+0000 7f76e6924640 1 -- 172.21.15.189:0/3996222656 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c0005080 con 0x7f76e01db110 2024-03-29T10:44:35.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.388+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f76d8006690 con 0x7f76e01db110 2024-03-29T10:44:35.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.388+0000 7f76dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:35.394 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.388+0000 7f76dcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:35.395 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:35.401+0000 7f3732c0f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:44:35.397 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:35.414+0000 7f546422c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:44:35.425 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:35.443+0000 7f3732c0f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:44:35.441 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:35.458+0000 7f546422c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:44:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.461+0000 7f76e6924640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30", "id": 3}] 2024-03-29T10:44:35.443 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.461+0000 7f76e6924640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30", "id": 3}] 2024-03-29T10:44:35.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.461+0000 7f76e6924640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.444 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.461+0000 7f76e6924640 1 -- 172.21.15.189:0/3996222656 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30", "id": 3} v 0) v1 -- 0x7f76c00058b0 con 0x7f76e01db110 2024-03-29T10:44:35.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.465+0000 7f76dcff9640 1 -- 172.21.15.189:0/3996222656 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3989830389 0 525046306) 0x7f76d8005960 con 0x7f76e01db110 2024-03-29T10:44:35.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.465+0000 7f76dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30", "id": 3}] 2024-03-29T10:44:35.447 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.465+0000 7f76dcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:35.448 INFO:teuthology.orchestra.run.smithi189.stdout:3 2024-03-29T10:44:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.465+0000 7f76e6924640 10 librados: watch_flush enter 2024-03-29T10:44:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.465+0000 7f76e6924640 10 librados: watch_flush exit 2024-03-29T10:44:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.465+0000 7f76e6924640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T10:44:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.465+0000 7f76e6924640 10 client.4129.objecter successfully canceled tick 2024-03-29T10:44:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.466+0000 7f76e6924640 10 monclient: shutdown 2024-03-29T10:44:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.466+0000 7f76e6924640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.466+0000 7f76e6924640 1 -- 172.21.15.189:0/3996222656 >> v1:172.21.15.189:6789/0 conn(0x7f76e01db110 legacy=0x7f76e0138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.449 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.466+0000 7f76e6924640 1 -- 172.21.15.189:0/3996222656 shutdown_connections 2024-03-29T10:44:35.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.466+0000 7f76e6924640 1 -- 172.21.15.189:0/3996222656 >> 172.21.15.189:0/3996222656 conn(0x7f76e0066b00 msgr2=0x7f76e013cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:35.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.466+0000 7f76e6924640 1 -- 172.21.15.189:0/3996222656 shutdown_connections 2024-03-29T10:44:35.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.466+0000 7f76e6924640 1 -- 172.21.15.189:0/3996222656 wait complete. 2024-03-29T10:44:35.450 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.466+0000 7f76e6924640 1 librados: shutdown 2024-03-29T10:44:35.459 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 4d6b02fb-0c49-4df6-9d91-c1600d8d6609 4 2024-03-29T10:44:35.541 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:35.558+0000 7f3732c0f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:44:35.563 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:35.580+0000 7f546422c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:44:35.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.617+0000 7f805bdf6640 10 monclient: get_monmap_and_config 2024-03-29T10:44:35.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.617+0000 7f805bdf6640 10 monclient: build_initial_monmap 2024-03-29T10:44:35.600 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.617+0000 7f805bdf6640 10 monclient: monmap: 2024-03-29T10:44:35.600 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:35.600 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.600 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:44:35.618047+0000 2024-03-29T10:44:35.600 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:44:35.618047+0000 2024-03-29T10:44:35.600 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:35.601 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.601 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:35.601 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:35.601 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:35.601 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.601 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.617+0000 7f805bdf6640 1 Processor -- start 2024-03-29T10:44:35.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.617+0000 7f805bdf6640 1 -- start start 2024-03-29T10:44:35.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.617+0000 7f805bdf6640 10 monclient: init 2024-03-29T10:44:35.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f805bdf6640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:35.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f805bdf6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:35.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f805bdf6640 10 monclient(hunting): picked mon.noname-b con 0x7f805409ac40 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:35.612 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f805bdf6640 10 monclient(hunting): picked mon.noname-a con 0x7f805409c870 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:35.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f805bdf6640 10 monclient(hunting): picked mon.noname-c con 0x7f80540b4360 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:35.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f805bdf6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805409dec0 con 0x7f805409c870 2024-03-29T10:44:35.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f805bdf6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805409da30 con 0x7f80540b4360 2024-03-29T10:44:35.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f805bdf6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80540b39c0 con 0x7f805409ac40 2024-03-29T10:44:35.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f805bdf6640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:35.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f805bdf6640 10 monclient(hunting): authenticate will time out at 885.912551s 2024-03-29T10:44:35.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f805a393640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f80540b4360 0x7f805414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57182/0 (socket says 172.21.15.189:57182) 2024-03-29T10:44:35.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f805a393640 1 -- 172.21.15.189:0/3859238671 learned_addr learned my addr 172.21.15.189:0/3859238671 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:35.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f8058b90640 1 -- 172.21.15.189:0/3859238671 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371638735 0 0) 0x7f805409da30 con 0x7f80540b4360 2024-03-29T10:44:35.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f8058b90640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f8058b90640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f8058b90640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.613 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f8058b90640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.618+0000 7f8058b90640 1 -- 172.21.15.189:0/3859238671 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8040003520 con 0x7f80540b4360 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 1 -- 172.21.15.189:0/3859238671 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354490867 0 0) 0x7f80540b39c0 con 0x7f805409ac40 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 1 -- 172.21.15.189:0/3859238671 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f805409da30 con 0x7f805409ac40 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 1 -- 172.21.15.189:0/3859238671 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847529316 0 0) 0x7f8050001040 con 0x7f805409c870 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 1 -- 172.21.15.189:0/3859238671 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80540b39c0 con 0x7f805409c870 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 1 -- 172.21.15.189:0/3859238671 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2428092297 0 0) 0x7f8040003520 con 0x7f80540b4360 2024-03-29T10:44:35.614 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 1 -- 172.21.15.189:0/3859238671 >> v1:172.21.15.98:6790/0 conn(0x7f805409ac40 legacy=0x7f80540b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 1 -- 172.21.15.189:0/3859238671 >> v1:172.21.15.98:6789/0 conn(0x7f805409c870 legacy=0x7f805413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient: _finish_hunting 0 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 1 monclient: found mon.noname-c 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 1 -- 172.21.15.189:0/3859238671 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80540b5160 con 0x7f80540b4360 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient: _finish_auth 0 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient: _check_auth_tickets 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 1 -- 172.21.15.189:0/3859238671 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f804c001a10 con 0x7f80540b4360 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f805bdf6640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f805bdf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.615 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient: dump: 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f805bdf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 1 -- 172.21.15.189:0/3859238671 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f804c001bf0 con 0x7f80540b4360 2024-03-29T10:44:35.616 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f805bdf6640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 1 -- 172.21.15.189:0/3859238671 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f804c002000 con 0x7f80540b4360 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f805bdf6640 4 set_mon_vals no callback set 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f8058b90640 10 monclient: dump: 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.617 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f805bdf6640 10 monclient: shutdown 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f805bdf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f805bdf6640 1 -- 172.21.15.189:0/3859238671 >> v1:172.21.15.189:6789/0 conn(0x7f80540b4360 legacy=0x7f805414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f805bdf6640 1 -- 172.21.15.189:0/3859238671 shutdown_connections 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f805bdf6640 1 -- 172.21.15.189:0/3859238671 >> 172.21.15.189:0/3859238671 conn(0x7f8054066b00 msgr2=0x7f8054066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f805bdf6640 1 -- 172.21.15.189:0/3859238671 shutdown_connections 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.619+0000 7f805bdf6640 1 -- 172.21.15.189:0/3859238671 wait complete. 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.620+0000 7f805bdf6640 10 monclient: build_initial_monmap 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.620+0000 7f805bdf6640 10 monclient: monmap: 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-03-29T10:44:35.618 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.620+0000 7f805bdf6640 1 librados: starting msgr at 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.620+0000 7f805bdf6640 1 librados: starting objecter 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.620+0000 7f805bdf6640 1 Processor -- start 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.620+0000 7f805bdf6640 1 -- start start 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.620+0000 7f805bdf6640 1 librados: setting wanted keys 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.620+0000 7f805bdf6640 1 librados: calling monclient init 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.620+0000 7f805bdf6640 10 monclient: init 2024-03-29T10:44:35.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f805bdf6640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:35.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f805bdf6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:35.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f805bdf6640 10 monclient(hunting): picked mon.noname-a con 0x7f805409ac40 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:35.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f805bdf6640 10 monclient(hunting): picked mon.noname-b con 0x7f80541d8690 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:35.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f805bdf6640 10 monclient(hunting): picked mon.noname-c con 0x7f80541455c0 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:35.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f805bdf6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80541d7af0 con 0x7f805409ac40 2024-03-29T10:44:35.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f805bdf6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8054145a10 con 0x7f80541d8690 2024-03-29T10:44:35.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f805bdf6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80540b5a20 con 0x7f80541455c0 2024-03-29T10:44:35.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f805bdf6640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:35.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f805bdf6640 10 monclient(hunting): authenticate will time out at 885.915517s 2024-03-29T10:44:35.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f8059391640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f80541d8690 0x7f8054144eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57190/0 (socket says 172.21.15.189:57190) 2024-03-29T10:44:35.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f8059391640 1 -- 172.21.15.189:0/299036207 learned_addr learned my addr 172.21.15.189:0/299036207 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:35.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 client.?.objecter ms_handle_connect 0x7f80541d8690 2024-03-29T10:44:35.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:35.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 client.?.objecter ms_handle_connect 0x7f805409ac40 2024-03-29T10:44:35.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:35.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 client.?.objecter ms_handle_connect 0x7f80541455c0 2024-03-29T10:44:35.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:35.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890449152 0 0) 0x7f8054145a10 con 0x7f80541d8690 2024-03-29T10:44:35.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:44:35.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8038004eb0 con 0x7f80541d8690 2024-03-29T10:44:35.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868008123 0 0) 0x7f80541d7af0 con 0x7f805409ac40 2024-03-29T10:44:35.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:44:35.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8054145a10 con 0x7f805409ac40 2024-03-29T10:44:35.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857103297 0 0) 0x7f80540b5a20 con 0x7f80541455c0 2024-03-29T10:44:35.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:44:35.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.621+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80541d7af0 con 0x7f80541455c0 2024-03-29T10:44:35.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3011664392 0 0) 0x7f8038004eb0 con 0x7f80541d8690 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80540b5a20 con 0x7f80541d8690 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8050001810 con 0x7f80541d8690 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient(hunting): dump: 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.623 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1491049913 0 0) 0x7f80541d7af0 con 0x7f80541455c0 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8038004eb0 con 0x7f80541455c0 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 717434533 0 0) 0x7f8054145a10 con 0x7f805409ac40 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80541d7af0 con 0x7f805409ac40 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f804c001d60 con 0x7f80541455c0 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient(hunting): dump: 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.624 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f80440018b0 con 0x7f805409ac40 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient(hunting): dump: 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.625 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.626 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.626 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.626 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.626 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.626 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2738386913 0 0) 0x7f80540b5a20 con 0x7f80541d8690 2024-03-29T10:44:35.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 >> v1:172.21.15.98:6790/0 conn(0x7f80541455c0 legacy=0x7f80541e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 >> v1:172.21.15.98:6789/0 conn(0x7f805409ac40 legacy=0x7f80541d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:35.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 monclient: found mon.b 2024-03-29T10:44:35.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:35.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.626 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8054141960 con 0x7f80541d8690 2024-03-29T10:44:35.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:35.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:35.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:35.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f805bdf6640 5 monclient: authenticate success, global_id 4135 2024-03-29T10:44:35.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f805bdf6640 10 monclient: _renew_subs 2024-03-29T10:44:35.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f805bdf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f805bdf6640 1 -- 172.21.15.189:0/299036207 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80541e3b20 con 0x7f80541d8690 2024-03-29T10:44:35.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f805bdf6640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:35.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f805bdf6640 10 monclient: _renew_subs 2024-03-29T10:44:35.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f805bdf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f805bdf6640 1 -- 172.21.15.189:0/299036207 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80541e3e90 con 0x7f80541d8690 2024-03-29T10:44:35.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f805bdf6640 1 librados: init done 2024-03-29T10:44:35.627 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8050003610 con 0x7f80541d8690 2024-03-29T10:44:35.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:35.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8050001d80 con 0x7f80541d8690 2024-03-29T10:44:35.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:35.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804bfff640 4 set_mon_vals no callback set 2024-03-29T10:44:35.628 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 monclient: dump: 2024-03-29T10:44:35.628 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.628 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.628 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.628 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.628 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.629 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.629 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.629 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.629 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.629 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8050003db0 con 0x7f80541d8690 2024-03-29T10:44:35.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 441602990 0 0) 0x7f80500057c0 con 0x7f80541d8690 2024-03-29T10:44:35.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 10 client.4135.objecter ms_dispatch 0x7f8054000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:44:35.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:44:35.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:44:35.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.622+0000 7f804a7fc640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T10:44:35.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.623+0000 7f805bdf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:35.629 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.623+0000 7f805bdf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:35.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.623+0000 7f805bdf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.623+0000 7f805bdf6640 1 -- 172.21.15.189:0/299036207 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803c005080 con 0x7f80541d8690 2024-03-29T10:44:35.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.625+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8050005a00 con 0x7f80541d8690 2024-03-29T10:44:35.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.625+0000 7f804a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:35.630 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.625+0000 7f804a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:35.630 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:35.627+0000 7f3732c0f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:44:35.637 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:35.654+0000 7f546422c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:44:35.655 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:35.673+0000 7f3732c0f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:44:35.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.697+0000 7f805bdf6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4d6b02fb-0c49-4df6-9d91-c1600d8d6609", "id": 4}] 2024-03-29T10:44:35.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.697+0000 7f805bdf6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4d6b02fb-0c49-4df6-9d91-c1600d8d6609", "id": 4}] 2024-03-29T10:44:35.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.697+0000 7f805bdf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.680 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.697+0000 7f805bdf6640 1 -- 172.21.15.189:0/299036207 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4d6b02fb-0c49-4df6-9d91-c1600d8d6609", "id": 4} v 0) v1 -- 0x7f803c0058b0 con 0x7f80541d8690 2024-03-29T10:44:35.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.701+0000 7f804a7fc640 1 -- 172.21.15.189:0/299036207 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4d6b02fb-0c49-4df6-9d91-c1600d8d6609", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 129545744 0 1697012583) 0x7f8050004060 con 0x7f80541d8690 2024-03-29T10:44:35.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.701+0000 7f804a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4d6b02fb-0c49-4df6-9d91-c1600d8d6609", "id": 4}] 2024-03-29T10:44:35.684 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.701+0000 7f804a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:35.684 INFO:teuthology.orchestra.run.smithi189.stdout:4 2024-03-29T10:44:35.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.702+0000 7f805bdf6640 10 librados: watch_flush enter 2024-03-29T10:44:35.685 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.702+0000 7f805bdf6640 10 librados: watch_flush exit 2024-03-29T10:44:35.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.702+0000 7f805bdf6640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T10:44:35.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.702+0000 7f805bdf6640 10 client.4135.objecter successfully canceled tick 2024-03-29T10:44:35.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.702+0000 7f805bdf6640 10 monclient: shutdown 2024-03-29T10:44:35.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.702+0000 7f805bdf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:35.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.702+0000 7f805bdf6640 1 -- 172.21.15.189:0/299036207 >> v1:172.21.15.189:6789/0 conn(0x7f80541d8690 legacy=0x7f8054144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.702+0000 7f805bdf6640 1 -- 172.21.15.189:0/299036207 shutdown_connections 2024-03-29T10:44:35.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.702+0000 7f805bdf6640 1 -- 172.21.15.189:0/299036207 >> 172.21.15.189:0/299036207 conn(0x7f8054066b00 msgr2=0x7f805413a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:35.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.702+0000 7f805bdf6640 1 -- 172.21.15.189:0/299036207 shutdown_connections 2024-03-29T10:44:35.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.702+0000 7f805bdf6640 1 -- 172.21.15.189:0/299036207 wait complete. 2024-03-29T10:44:35.686 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.702+0000 7f805bdf6640 1 librados: shutdown 2024-03-29T10:44:35.686 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:35.702+0000 7f546422c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:44:35.695 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 1083e880-407b-4f66-9207-c62e48fd52f5 5 2024-03-29T10:44:35.709 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:35.726+0000 7f3732c0f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:44:35.743 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:35.760+0000 7f546422c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:44:35.756 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:35.773+0000 7f3732c0f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:44:35.793 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:35.810+0000 7f546422c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:44:35.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.813+0000 7f5f3c03e640 10 monclient: get_monmap_and_config 2024-03-29T10:44:35.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.813+0000 7f5f3c03e640 10 monclient: build_initial_monmap 2024-03-29T10:44:35.796 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.813+0000 7f5f3c03e640 10 monclient: monmap: 2024-03-29T10:44:35.796 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:35.796 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.796 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:44:35.814627+0000 2024-03-29T10:44:35.796 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:44:35.814627+0000 2024-03-29T10:44:35.796 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:35.797 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.797 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:35.797 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:35.797 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:35.797 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.814+0000 7f5f3c03e640 1 Processor -- start 2024-03-29T10:44:35.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.814+0000 7f5f3c03e640 1 -- start start 2024-03-29T10:44:35.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.814+0000 7f5f3c03e640 10 monclient: init 2024-03-29T10:44:35.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.814+0000 7f5f3c03e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:35.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.814+0000 7f5f3c03e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:35.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.814+0000 7f5f3c03e640 10 monclient(hunting): picked mon.noname-a con 0x7f5f340b1780 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.814+0000 7f5f3c03e640 10 monclient(hunting): picked mon.noname-c con 0x7f5f3413d7e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f3c03e640 10 monclient(hunting): picked mon.noname-b con 0x7f5f34143be0 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f3c03e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f34070b30 con 0x7f5f340b1780 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f3c03e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f34070da0 con 0x7f5f3413d7e0 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f3c03e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3409adb0 con 0x7f5f34143be0 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f3c03e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f3c03e640 10 monclient(hunting): authenticate will time out at 886.108991s 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f395d9640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5f3413d7e0 0x7f5f341434d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57198/0 (socket says 172.21.15.189:57198) 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f395d9640 1 -- 172.21.15.189:0/1230531338 learned_addr learned my addr 172.21.15.189:0/1230531338 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 -- 172.21.15.189:0/1230531338 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089463166 0 0) 0x7f5f34070da0 con 0x7f5f3413d7e0 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.807 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 -- 172.21.15.189:0/1230531338 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f20003520 con 0x7f5f3413d7e0 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 -- 172.21.15.189:0/1230531338 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876929980 0 0) 0x7f5f34070b30 con 0x7f5f340b1780 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 -- 172.21.15.189:0/1230531338 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f34070da0 con 0x7f5f340b1780 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 -- 172.21.15.189:0/1230531338 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618982161 0 0) 0x7f5f3409adb0 con 0x7f5f34143be0 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 -- 172.21.15.189:0/1230531338 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f34070b30 con 0x7f5f34143be0 2024-03-29T10:44:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 -- 172.21.15.189:0/1230531338 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 245208776 0 0) 0x7f5f20003520 con 0x7f5f3413d7e0 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 -- 172.21.15.189:0/1230531338 >> v1:172.21.15.98:6790/0 conn(0x7f5f34143be0 legacy=0x7f5f3414b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 -- 172.21.15.189:0/1230531338 >> v1:172.21.15.98:6789/0 conn(0x7f5f340b1780 legacy=0x7f5f3413d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient: _finish_hunting 0 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 monclient: found mon.noname-c 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 -- 172.21.15.189:0/1230531338 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f34144070 con 0x7f5f3413d7e0 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient: _finish_auth 0 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient: _check_auth_tickets 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 -- 172.21.15.189:0/1230531338 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5f24001690 con 0x7f5f3413d7e0 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f3c03e640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f3c03e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient: dump: 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f3c03e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 -- 172.21.15.189:0/1230531338 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f24001870 con 0x7f5f3413d7e0 2024-03-29T10:44:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 1 -- 172.21.15.189:0/1230531338 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5f24001c80 con 0x7f5f3413d7e0 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f38dd8640 10 monclient: dump: 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f3c03e640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.815+0000 7f5f3c03e640 4 set_mon_vals no callback set 2024-03-29T10:44:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.816+0000 7f5f3c03e640 10 monclient: shutdown 2024-03-29T10:44:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.816+0000 7f5f3c03e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.816+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1230531338 >> v1:172.21.15.189:6789/0 conn(0x7f5f3413d7e0 legacy=0x7f5f341434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.816+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1230531338 shutdown_connections 2024-03-29T10:44:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.816+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1230531338 >> 172.21.15.189:0/1230531338 conn(0x7f5f340b58f0 msgr2=0x7f5f340b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.816+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1230531338 shutdown_connections 2024-03-29T10:44:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.816+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1230531338 wait complete. 2024-03-29T10:44:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.817+0000 7f5f3c03e640 10 monclient: build_initial_monmap 2024-03-29T10:44:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.817+0000 7f5f3c03e640 10 monclient: monmap: 2024-03-29T10:44:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-03-29T10:44:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-03-29T10:44:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:35.813 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.817+0000 7f5f3c03e640 1 librados: starting msgr at 2024-03-29T10:44:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.817+0000 7f5f3c03e640 1 librados: starting objecter 2024-03-29T10:44:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.817+0000 7f5f3c03e640 1 Processor -- start 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.817+0000 7f5f3c03e640 1 -- start start 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.817+0000 7f5f3c03e640 1 librados: setting wanted keys 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.817+0000 7f5f3c03e640 1 librados: calling monclient init 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.817+0000 7f5f3c03e640 10 monclient: init 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 10 monclient(hunting): picked mon.noname-a con 0x7f5f340b1780 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 10 monclient(hunting): picked mon.noname-b con 0x7f5f3413d7e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 10 monclient(hunting): picked mon.noname-c con 0x7f5f34143be0 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3413ee90 con 0x7f5f340b1780 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f34139af0 con 0x7f5f3413d7e0 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f34139d20 con 0x7f5f34143be0 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 10 monclient(hunting): authenticate will time out at 886.112073s 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f395d9640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5f3413d7e0 0x7f5f34134990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57214/0 (socket says 172.21.15.189:57214) 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f395d9640 1 -- 172.21.15.189:0/1260451361 learned_addr learned my addr 172.21.15.189:0/1260451361 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 client.?.objecter ms_handle_connect 0x7f5f3413d7e0 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 client.?.objecter ms_handle_connect 0x7f5f340b1780 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 client.?.objecter ms_handle_connect 0x7f5f34143be0 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477992184 0 0) 0x7f5f34139af0 con 0x7f5f3413d7e0 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f18004eb0 con 0x7f5f3413d7e0 2024-03-29T10:44:35.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780110635 0 0) 0x7f5f3413ee90 con 0x7f5f340b1780 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f34139af0 con 0x7f5f340b1780 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567892618 0 0) 0x7f5f34139d20 con 0x7f5f34143be0 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f3413ee90 con 0x7f5f34143be0 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2150090897 0 0) 0x7f5f34139af0 con 0x7f5f340b1780 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f34139d20 con 0x7f5f340b1780 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5f30001930 con 0x7f5f340b1780 2024-03-29T10:44:35.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:35.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): dump: 2024-03-29T10:44:35.817 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.817 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.817 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.817 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.817 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.817 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.817 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.817 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.817 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.817 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:35.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2207319045 0 0) 0x7f5f18004eb0 con 0x7f5f3413d7e0 2024-03-29T10:44:35.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f34139af0 con 0x7f5f3413d7e0 2024-03-29T10:44:35.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5f24001bc0 con 0x7f5f3413d7e0 2024-03-29T10:44:35.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:35.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): dump: 2024-03-29T10:44:35.818 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.818 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.818 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.818 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.819 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.819 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.819 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.819 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.819 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.819 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2786920004 0 0) 0x7f5f3413ee90 con 0x7f5f34143be0 2024-03-29T10:44:35.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f18004eb0 con 0x7f5f34143be0 2024-03-29T10:44:35.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5f2c001b50 con 0x7f5f34143be0 2024-03-29T10:44:35.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:35.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient(hunting): dump: 2024-03-29T10:44:35.819 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.820 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.820 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.820 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.820 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.820 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.820 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.820 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.820 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.820 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1135165875 0 0) 0x7f5f34139d20 con 0x7f5f340b1780 2024-03-29T10:44:35.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 >> v1:172.21.15.98:6790/0 conn(0x7f5f34143be0 legacy=0x7f5f341e0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 >> v1:172.21.15.189:6789/0 conn(0x7f5f3413d7e0 legacy=0x7f5f34134990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:35.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 monclient: found mon.a 2024-03-29T10:44:35.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:35.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:35.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3413a000 con 0x7f5f340b1780 2024-03-29T10:44:35.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:35.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:35.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:35.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 5 monclient: authenticate success, global_id 4149 2024-03-29T10:44:35.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 10 monclient: _renew_subs 2024-03-29T10:44:35.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:35.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1260451361 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f34138840 con 0x7f5f340b1780 2024-03-29T10:44:35.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:35.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 10 monclient: _renew_subs 2024-03-29T10:44:35.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:35.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1260451361 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f34137e10 con 0x7f5f340b1780 2024-03-29T10:44:35.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.818+0000 7f5f3c03e640 1 librados: init done 2024-03-29T10:44:35.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f300035f0 con 0x7f5f340b1780 2024-03-29T10:44:35.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:35.822 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5f30003900 con 0x7f5f340b1780 2024-03-29T10:44:35.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:35.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f2a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:35.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f2bfff640 4 set_mon_vals no callback set 2024-03-29T10:44:35.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f2a7fc640 10 monclient: dump: 2024-03-29T10:44:35.823 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:35.823 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.823 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.823 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:35.823 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:35.823 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.823 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:35.824 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:35.824 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:35.824 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:35.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5f30003df0 con 0x7f5f340b1780 2024-03-29T10:44:35.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3820569184 0 0) 0x7f5f30005560 con 0x7f5f340b1780 2024-03-29T10:44:35.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f2a7fc640 10 client.4149.objecter ms_dispatch 0x7f5f34000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:44:35.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f2a7fc640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:44:35.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f2a7fc640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:44:35.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f2a7fc640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-29T10:44:35.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f3c03e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:35.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f3c03e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:35.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f3c03e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:35.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.819+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1260451361 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f1c005080 con 0x7f5f340b1780 2024-03-29T10:44:35.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.821+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5f300050f0 con 0x7f5f340b1780 2024-03-29T10:44:35.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.821+0000 7f5f2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:35.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.821+0000 7f5f2a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:35.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.894+0000 7f5f3c03e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1083e880-407b-4f66-9207-c62e48fd52f5", "id": 5}] 2024-03-29T10:44:35.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.894+0000 7f5f3c03e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1083e880-407b-4f66-9207-c62e48fd52f5", "id": 5}] 2024-03-29T10:44:35.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.894+0000 7f5f3c03e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:35.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.894+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1260451361 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1083e880-407b-4f66-9207-c62e48fd52f5", "id": 5} v 0) v1 -- 0x7f5f1c0058b0 con 0x7f5f340b1780 2024-03-29T10:44:35.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.895+0000 7f5f2a7fc640 1 -- 172.21.15.189:0/1260451361 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1083e880-407b-4f66-9207-c62e48fd52f5", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2655751311 0 1988412176) 0x7f5f30005360 con 0x7f5f340b1780 2024-03-29T10:44:35.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.895+0000 7f5f2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1083e880-407b-4f66-9207-c62e48fd52f5", "id": 5}] 2024-03-29T10:44:35.878 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.895+0000 7f5f2a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:35.879 INFO:teuthology.orchestra.run.smithi189.stdout:5 2024-03-29T10:44:35.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.896+0000 7f5f3c03e640 10 librados: watch_flush enter 2024-03-29T10:44:35.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.896+0000 7f5f3c03e640 10 librados: watch_flush exit 2024-03-29T10:44:35.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.896+0000 7f5f3c03e640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-29T10:44:35.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.896+0000 7f5f3c03e640 10 client.4149.objecter successfully canceled tick 2024-03-29T10:44:35.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.896+0000 7f5f3c03e640 10 monclient: shutdown 2024-03-29T10:44:35.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.896+0000 7f5f3c03e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:35.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.896+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1260451361 >> v1:172.21.15.98:6789/0 conn(0x7f5f340b1780 legacy=0x7f5f341d52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.896+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1260451361 shutdown_connections 2024-03-29T10:44:35.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.896+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1260451361 >> 172.21.15.189:0/1260451361 conn(0x7f5f340b58f0 msgr2=0x7f5f34069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:35.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.896+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1260451361 shutdown_connections 2024-03-29T10:44:35.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.896+0000 7f5f3c03e640 1 -- 172.21.15.189:0/1260451361 wait complete. 2024-03-29T10:44:35.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:35.896+0000 7f5f3c03e640 1 librados: shutdown 2024-03-29T10:44:35.890 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new cfeca019-a949-4d0d-aafc-afd40c7f39fa 6 2024-03-29T10:44:35.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.007+0000 7fa7e4923640 10 monclient: get_monmap_and_config 2024-03-29T10:44:35.990 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.007+0000 7fa7e4923640 10 monclient: build_initial_monmap 2024-03-29T10:44:35.991 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.007+0000 7fa7e4923640 10 monclient: monmap: 2024-03-29T10:44:35.991 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:35.991 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:35.991 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:44:36.008807+0000 2024-03-29T10:44:35.991 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:44:36.008807+0000 2024-03-29T10:44:35.991 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:35.991 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:35.991 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:35.991 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:35.991 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:35.991 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.008+0000 7fa7e4923640 1 Processor -- start 2024-03-29T10:44:36.001 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.008+0000 7fa7e4923640 1 -- start start 2024-03-29T10:44:36.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.008+0000 7fa7e4923640 10 monclient: init 2024-03-29T10:44:36.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7e4923640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:36.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7e4923640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:36.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7e4923640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e009ac40 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:36.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7e4923640 10 monclient(hunting): picked mon.noname-b con 0x7fa7e009c870 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:36.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7e4923640 10 monclient(hunting): picked mon.noname-a con 0x7fa7e00b4360 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:36.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7e4923640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e009dec0 con 0x7fa7e00b4360 2024-03-29T10:44:36.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7e4923640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e009da30 con 0x7fa7e009ac40 2024-03-29T10:44:36.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7e4923640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e00b39c0 con 0x7fa7e009c870 2024-03-29T10:44:36.002 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7e4923640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:36.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7e4923640 10 monclient(hunting): authenticate will time out at 886.303192s 2024-03-29T10:44:36.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7de59c640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa7e009ac40 0x7fa7e00b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57216/0 (socket says 172.21.15.189:57216) 2024-03-29T10:44:36.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7de59c640 1 -- 172.21.15.189:0/4183229106 learned_addr learned my addr 172.21.15.189:0/4183229106 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:36.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 1 -- 172.21.15.189:0/4183229106 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385458534 0 0) 0x7fa7e009da30 con 0x7fa7e009ac40 2024-03-29T10:44:36.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:44:36.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 1 -- 172.21.15.189:0/4183229106 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c8003520 con 0x7fa7e009ac40 2024-03-29T10:44:36.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 1 -- 172.21.15.189:0/4183229106 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394274157 0 0) 0x7fa7d8001490 con 0x7fa7e00b4360 2024-03-29T10:44:36.003 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:44:36.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 1 -- 172.21.15.189:0/4183229106 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e009da30 con 0x7fa7e00b4360 2024-03-29T10:44:36.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 1 -- 172.21.15.189:0/4183229106 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145241385 0 0) 0x7fa7e00b39c0 con 0x7fa7e009c870 2024-03-29T10:44:36.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:44:36.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 1 -- 172.21.15.189:0/4183229106 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d8001490 con 0x7fa7e009c870 2024-03-29T10:44:36.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 1 -- 172.21.15.189:0/4183229106 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2580037451 0 0) 0x7fa7c8003520 con 0x7fa7e009ac40 2024-03-29T10:44:36.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 1 -- 172.21.15.189:0/4183229106 >> v1:172.21.15.98:6790/0 conn(0x7fa7e009c870 legacy=0x7fa7e013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.004 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 1 -- 172.21.15.189:0/4183229106 >> v1:172.21.15.98:6789/0 conn(0x7fa7e00b4360 legacy=0x7fa7e014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient: _finish_hunting 0 2024-03-29T10:44:36.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 1 monclient: found mon.noname-c 2024-03-29T10:44:36.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:36.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:44:36.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 1 -- 172.21.15.189:0/4183229106 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e00b5160 con 0x7fa7e009ac40 2024-03-29T10:44:36.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient: _finish_auth 0 2024-03-29T10:44:36.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient: _check_auth_tickets 2024-03-29T10:44:36.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:36.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7e4923640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:44:36.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 1 -- 172.21.15.189:0/4183229106 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa7d4001690 con 0x7fa7e009ac40 2024-03-29T10:44:36.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7e4923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:36.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:36.006 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7dd59a640 10 monclient: dump: 2024-03-29T10:44:36.006 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:36.006 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.006 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.006 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.006 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.006 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.006 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.006 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.006 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.006 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.009+0000 7fa7e4923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7dd59a640 1 -- 172.21.15.189:0/4183229106 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7d4001870 con 0x7fa7e009ac40 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7dd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7dd59a640 1 -- 172.21.15.189:0/4183229106 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa7d4001c80 con 0x7fa7e009ac40 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7e4923640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7e4923640 4 set_mon_vals no callback set 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7dd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7dd59a640 10 monclient: dump: 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.008 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7e4923640 10 monclient: shutdown 2024-03-29T10:44:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7e4923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7e4923640 1 -- 172.21.15.189:0/4183229106 >> v1:172.21.15.189:6789/0 conn(0x7fa7e009ac40 legacy=0x7fa7e00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7e4923640 1 -- 172.21.15.189:0/4183229106 shutdown_connections 2024-03-29T10:44:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7e4923640 1 -- 172.21.15.189:0/4183229106 >> 172.21.15.189:0/4183229106 conn(0x7fa7e0066b00 msgr2=0x7fa7e0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7e4923640 1 -- 172.21.15.189:0/4183229106 shutdown_connections 2024-03-29T10:44:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.010+0000 7fa7e4923640 1 -- 172.21.15.189:0/4183229106 wait complete. 2024-03-29T10:44:36.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.011+0000 7fa7e4923640 10 monclient: build_initial_monmap 2024-03-29T10:44:36.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.011+0000 7fa7e4923640 10 monclient: monmap: 2024-03-29T10:44:36.009 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:36.009 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:36.009 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-03-29T10:44:36.009 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-03-29T10:44:36.009 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:36.009 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.009 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:36.009 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:36.009 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:36.009 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.009 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.011+0000 7fa7e4923640 1 librados: starting msgr at 2024-03-29T10:44:36.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.011+0000 7fa7e4923640 1 librados: starting objecter 2024-03-29T10:44:36.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.011+0000 7fa7e4923640 1 Processor -- start 2024-03-29T10:44:36.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.011+0000 7fa7e4923640 1 -- start start 2024-03-29T10:44:36.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.011+0000 7fa7e4923640 1 librados: setting wanted keys 2024-03-29T10:44:36.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.011+0000 7fa7e4923640 1 librados: calling monclient init 2024-03-29T10:44:36.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.011+0000 7fa7e4923640 10 monclient: init 2024-03-29T10:44:36.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7e4923640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:36.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7e4923640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:36.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7e4923640 10 monclient(hunting): picked mon.noname-a con 0x7fa7e009ac40 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:36.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7e4923640 10 monclient(hunting): picked mon.noname-c con 0x7fa7e00b0210 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:36.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7e4923640 10 monclient(hunting): picked mon.noname-b con 0x7fa7e01d7c90 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:36.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7e4923640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e00afb50 con 0x7fa7e009ac40 2024-03-29T10:44:36.010 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7e4923640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e01d70f0 con 0x7fa7e01d7c90 2024-03-29T10:44:36.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7e4923640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e00b0650 con 0x7fa7e00b0210 2024-03-29T10:44:36.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7e4923640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:36.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7e4923640 10 monclient(hunting): authenticate will time out at 886.306329s 2024-03-29T10:44:36.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ded9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa7e01d7c90 0x7fa7e01e1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57232/0 (socket says 172.21.15.189:57232) 2024-03-29T10:44:36.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ded9d640 1 -- 172.21.15.189:0/355480603 learned_addr learned my addr 172.21.15.189:0/355480603 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:36.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 client.?.objecter ms_handle_connect 0x7fa7e01d7c90 2024-03-29T10:44:36.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:36.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 client.?.objecter ms_handle_connect 0x7fa7e009ac40 2024-03-29T10:44:36.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:36.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 client.?.objecter ms_handle_connect 0x7fa7e00b0210 2024-03-29T10:44:36.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:36.011 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375753502 0 0) 0x7fa7e01d70f0 con 0x7fa7e01d7c90 2024-03-29T10:44:36.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:44:36.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c0004eb0 con 0x7fa7e01d7c90 2024-03-29T10:44:36.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096002616 0 0) 0x7fa7e00afb50 con 0x7fa7e009ac40 2024-03-29T10:44:36.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:44:36.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e01d70f0 con 0x7fa7e009ac40 2024-03-29T10:44:36.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715292568 0 0) 0x7fa7e00b0650 con 0x7fa7e00b0210 2024-03-29T10:44:36.012 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e00afb50 con 0x7fa7e00b0210 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1156104947 0 0) 0x7fa7c0004eb0 con 0x7fa7e01d7c90 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7e00b0650 con 0x7fa7e01d7c90 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa7d8001bf0 con 0x7fa7e01d7c90 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.012+0000 7fa7ceffd640 10 monclient(hunting): dump: 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.013 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.014 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.014 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.014 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.014 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.014 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.014 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:36.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4083833757 0 0) 0x7fa7e00afb50 con 0x7fa7e00b0210 2024-03-29T10:44:36.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7c0004eb0 con 0x7fa7e00b0210 2024-03-29T10:44:36.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1869148343 0 0) 0x7fa7e01d70f0 con 0x7fa7e009ac40 2024-03-29T10:44:36.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7e00afb50 con 0x7fa7e009ac40 2024-03-29T10:44:36.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa7d0001830 con 0x7fa7e00b0210 2024-03-29T10:44:36.014 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:36.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient(hunting): dump: 2024-03-29T10:44:36.015 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:36.015 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.015 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.015 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.015 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.015 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.015 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.015 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.015 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.015 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa7d4001b80 con 0x7fa7e009ac40 2024-03-29T10:44:36.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:36.016 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient(hunting): dump: 2024-03-29T10:44:36.016 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:36.016 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.016 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.016 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.016 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.016 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.016 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.016 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.016 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.016 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1937968967 0 0) 0x7fa7e00b0650 con 0x7fa7e01d7c90 2024-03-29T10:44:36.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 >> v1:172.21.15.98:6790/0 conn(0x7fa7e00b0210 legacy=0x7fa7e01d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 >> v1:172.21.15.98:6789/0 conn(0x7fa7e009ac40 legacy=0x7fa7e00af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:36.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 monclient: found mon.b 2024-03-29T10:44:36.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:36.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:36.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e00acd00 con 0x7fa7e01d7c90 2024-03-29T10:44:36.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:36.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:36.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:36.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7e4923640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:44:36.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7e4923640 10 monclient: _renew_subs 2024-03-29T10:44:36.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7e4923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:36.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7e4923640 1 -- 172.21.15.189:0/355480603 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e00b18a0 con 0x7fa7e01d7c90 2024-03-29T10:44:36.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7e4923640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:36.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7e4923640 10 monclient: _renew_subs 2024-03-29T10:44:36.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7e4923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:36.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7e4923640 1 -- 172.21.15.189:0/355480603 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e01e35d0 con 0x7fa7e01d7c90 2024-03-29T10:44:36.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7e4923640 1 librados: init done 2024-03-29T10:44:36.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7d8003990 con 0x7fa7e01d7c90 2024-03-29T10:44:36.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:36.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa7d8003d20 con 0x7fa7e01d7c90 2024-03-29T10:44:36.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:36.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 monclient: dump: 2024-03-29T10:44:36.019 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:36.019 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.019 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.019 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.019 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.019 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.019 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.019 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.019 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.019 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7dcd99640 4 set_mon_vals no callback set 2024-03-29T10:44:36.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa7d8004290 con 0x7fa7e01d7c90 2024-03-29T10:44:36.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1746590151 0 0) 0x7fa7d8006b20 con 0x7fa7e01d7c90 2024-03-29T10:44:36.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 10 client.4147.objecter ms_dispatch 0x7fa7e0000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:44:36.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:44:36.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:44:36.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.013+0000 7fa7ceffd640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T10:44:36.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.014+0000 7fa7e4923640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:36.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.014+0000 7fa7e4923640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:36.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.014+0000 7fa7e4923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:36.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.014+0000 7fa7e4923640 1 -- 172.21.15.189:0/355480603 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e01d72a0 con 0x7fa7e01d7c90 2024-03-29T10:44:36.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.016+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa7e01d72a0 con 0x7fa7e01d7c90 2024-03-29T10:44:36.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.016+0000 7fa7ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:36.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.016+0000 7fa7ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:36.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.089+0000 7fa7e4923640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cfeca019-a949-4d0d-aafc-afd40c7f39fa", "id": 6}] 2024-03-29T10:44:36.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.089+0000 7fa7e4923640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cfeca019-a949-4d0d-aafc-afd40c7f39fa", "id": 6}] 2024-03-29T10:44:36.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.089+0000 7fa7e4923640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:36.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.089+0000 7fa7e4923640 1 -- 172.21.15.189:0/355480603 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd new", "uuid": "cfeca019-a949-4d0d-aafc-afd40c7f39fa", "id": 6} v 0) v1 -- 0x7fa7e004f730 con 0x7fa7e01d7c90 2024-03-29T10:44:36.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.092+0000 7fa7ceffd640 1 -- 172.21.15.189:0/355480603 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cfeca019-a949-4d0d-aafc-afd40c7f39fa", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3681838287 0 1113811849) 0x7fa7d8004800 con 0x7fa7e01d7c90 2024-03-29T10:44:36.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.092+0000 7fa7ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cfeca019-a949-4d0d-aafc-afd40c7f39fa", "id": 6}] 2024-03-29T10:44:36.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.092+0000 7fa7ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:36.075 INFO:teuthology.orchestra.run.smithi189.stdout:6 2024-03-29T10:44:36.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.093+0000 7fa7e4923640 10 librados: watch_flush enter 2024-03-29T10:44:36.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.093+0000 7fa7e4923640 10 librados: watch_flush exit 2024-03-29T10:44:36.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.093+0000 7fa7e4923640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T10:44:36.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.093+0000 7fa7e4923640 10 client.4147.objecter successfully canceled tick 2024-03-29T10:44:36.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.093+0000 7fa7e4923640 10 monclient: shutdown 2024-03-29T10:44:36.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.093+0000 7fa7e4923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:36.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.093+0000 7fa7e4923640 1 -- 172.21.15.189:0/355480603 >> v1:172.21.15.189:6789/0 conn(0x7fa7e01d7c90 legacy=0x7fa7e01e1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.093+0000 7fa7e4923640 1 -- 172.21.15.189:0/355480603 shutdown_connections 2024-03-29T10:44:36.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.093+0000 7fa7e4923640 1 -- 172.21.15.189:0/355480603 >> 172.21.15.189:0/355480603 conn(0x7fa7e0066b00 msgr2=0x7fa7e00b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:36.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.093+0000 7fa7e4923640 1 -- 172.21.15.189:0/355480603 shutdown_connections 2024-03-29T10:44:36.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.093+0000 7fa7e4923640 1 -- 172.21.15.189:0/355480603 wait complete. 2024-03-29T10:44:36.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.093+0000 7fa7e4923640 1 librados: shutdown 2024-03-29T10:44:36.086 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 2132347f-1364-46a0-9722-bf70172452ac 7 2024-03-29T10:44:36.139 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:36.156+0000 7f3732c0f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:44:36.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.204+0000 7f0e3e216640 10 monclient: get_monmap_and_config 2024-03-29T10:44:36.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.204+0000 7f0e3e216640 10 monclient: build_initial_monmap 2024-03-29T10:44:36.187 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.204+0000 7f0e3e216640 10 monclient: monmap: 2024-03-29T10:44:36.187 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:36.187 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.187 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:44:36.205244+0000 2024-03-29T10:44:36.188 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:44:36.205244+0000 2024-03-29T10:44:36.188 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:36.188 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.188 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:36.188 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:36.188 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:36.188 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e3e216640 1 Processor -- start 2024-03-29T10:44:36.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e3e216640 1 -- start start 2024-03-29T10:44:36.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e3e216640 10 monclient: init 2024-03-29T10:44:36.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e3e216640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:36.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e3e216640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:36.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e3e216640 10 monclient(hunting): picked mon.noname-c con 0x7f0e3809ac40 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:36.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e3e216640 10 monclient(hunting): picked mon.noname-b con 0x7f0e3809c870 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:36.197 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e3e216640 10 monclient(hunting): picked mon.noname-a con 0x7f0e380b4360 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e3e216640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3809dec0 con 0x7f0e380b4360 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e3e216640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3809da30 con 0x7f0e3809ac40 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e3e216640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e380b39c0 con 0x7f0e3809c870 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e3e216640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e3e216640 10 monclient(hunting): authenticate will time out at 886.499718s 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e377fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0e3809ac40 0x7f0e380b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57238/0 (socket says 172.21.15.189:57238) 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.205+0000 7f0e377fe640 1 -- 172.21.15.189:0/3050754114 learned_addr learned my addr 172.21.15.189:0/3050754114 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 -- 172.21.15.189:0/3050754114 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300457382 0 0) 0x7f0e3809da30 con 0x7f0e3809ac40 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 -- 172.21.15.189:0/3050754114 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e20003520 con 0x7f0e3809ac40 2024-03-29T10:44:36.198 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 -- 172.21.15.189:0/3050754114 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206164041 0 0) 0x7f0e30001490 con 0x7f0e380b4360 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient(hunting): my global_id is 4158 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 -- 172.21.15.189:0/3050754114 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e3809da30 con 0x7f0e380b4360 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 -- 172.21.15.189:0/3050754114 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307749443 0 0) 0x7f0e380b39c0 con 0x7f0e3809c870 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 -- 172.21.15.189:0/3050754114 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e30001490 con 0x7f0e3809c870 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 -- 172.21.15.189:0/3050754114 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3572948831 0 0) 0x7f0e20003520 con 0x7f0e3809ac40 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 -- 172.21.15.189:0/3050754114 >> v1:172.21.15.98:6790/0 conn(0x7f0e3809c870 legacy=0x7f0e3813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 -- 172.21.15.189:0/3050754114 >> v1:172.21.15.98:6789/0 conn(0x7f0e380b4360 legacy=0x7f0e3814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.199 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 monclient: found mon.noname-c 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 -- 172.21.15.189:0/3050754114 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e380b5160 con 0x7f0e3809ac40 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 -- 172.21.15.189:0/3050754114 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f0e2c001690 con 0x7f0e3809ac40 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient: dump: 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.200 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e3e216640 5 monclient: authenticate success, global_id 4150 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e3e216640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 -- 172.21.15.189:0/3050754114 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e2c001870 con 0x7f0e3809ac40 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 1 -- 172.21.15.189:0/3050754114 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f0e2c001c80 con 0x7f0e3809ac40 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e3e216640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e3e216640 4 set_mon_vals no callback set 2024-03-29T10:44:36.201 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.206+0000 7f0e367fc640 10 monclient: dump: 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.207+0000 7f0e3e216640 10 monclient: shutdown 2024-03-29T10:44:36.202 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.207+0000 7f0e3e216640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.207+0000 7f0e3e216640 1 -- 172.21.15.189:0/3050754114 >> v1:172.21.15.189:6789/0 conn(0x7f0e3809ac40 legacy=0x7f0e380b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.207+0000 7f0e3e216640 1 -- 172.21.15.189:0/3050754114 shutdown_connections 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.207+0000 7f0e3e216640 1 -- 172.21.15.189:0/3050754114 >> 172.21.15.189:0/3050754114 conn(0x7f0e38066b00 msgr2=0x7f0e38066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.207+0000 7f0e3e216640 1 -- 172.21.15.189:0/3050754114 shutdown_connections 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.207+0000 7f0e3e216640 1 -- 172.21.15.189:0/3050754114 wait complete. 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.207+0000 7f0e3e216640 10 monclient: build_initial_monmap 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.207+0000 7f0e3e216640 10 monclient: monmap: 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 0 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 0.000000 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:created 0.000000 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:36.203 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.207+0000 7f0e3e216640 1 librados: starting msgr at 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.207+0000 7f0e3e216640 1 librados: starting objecter 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 1 Processor -- start 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 1 -- start start 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 1 librados: setting wanted keys 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 1 librados: calling monclient init 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 10 monclient: init 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 10 monclient(hunting): picked mon.noname-c con 0x7f0e3809ac40 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 10 monclient(hunting): picked mon.noname-a con 0x7f0e380b1860 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 10 monclient(hunting): picked mon.noname-b con 0x7f0e3813b5b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e381d7660 con 0x7f0e380b1860 2024-03-29T10:44:36.204 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e380ade20 con 0x7f0e3813b5b0 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e381406b0 con 0x7f0e3809ac40 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e3e216640 10 monclient(hunting): authenticate will time out at 886.502843s 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e37fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0e3813b5b0 0x7f0e381e2830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:57252/0 (socket says 172.21.15.189:57252) 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.208+0000 7f0e37fff640 1 -- 172.21.15.189:0/2595279121 learned_addr learned my addr 172.21.15.189:0/2595279121 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 client.?.objecter ms_handle_connect 0x7f0e3813b5b0 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 client.?.objecter ms_handle_connect 0x7f0e3809ac40 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 client.?.objecter ms_handle_connect 0x7f0e380b1860 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628203215 0 0) 0x7f0e380ade20 con 0x7f0e3813b5b0 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.205 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e24004eb0 con 0x7f0e3813b5b0 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489345921 0 0) 0x7f0e381d7660 con 0x7f0e380b1860 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): my global_id is 4161 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e380ade20 con 0x7f0e380b1860 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059470087 0 0) 0x7f0e381406b0 con 0x7f0e3809ac40 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.206 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): my global_id is 4154 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e381d7660 con 0x7f0e3809ac40 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2917538034 0 0) 0x7f0e380ade20 con 0x7f0e380b1860 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e381406b0 con 0x7f0e380b1860 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f0e28001870 con 0x7f0e380b1860 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): dump: 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.207 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.208 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.208 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:36.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2433003551 0 0) 0x7f0e24004eb0 con 0x7f0e3813b5b0 2024-03-29T10:44:36.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e380ade20 con 0x7f0e3813b5b0 2024-03-29T10:44:36.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f0e30001b70 con 0x7f0e3813b5b0 2024-03-29T10:44:36.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:36.208 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): dump: 2024-03-29T10:44:36.208 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:36.208 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.208 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.208 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.209 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.209 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.209 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.209 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.209 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.209 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2669633365 0 0) 0x7f0e381d7660 con 0x7f0e3809ac40 2024-03-29T10:44:36.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e24004eb0 con 0x7f0e3809ac40 2024-03-29T10:44:36.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f0e2c001b80 con 0x7f0e3809ac40 2024-03-29T10:44:36.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:36.209 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient(hunting): dump: 2024-03-29T10:44:36.210 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:36.210 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.210 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.210 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.210 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.210 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.211 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.211 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.211 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.211 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3342806142 0 0) 0x7f0e381406b0 con 0x7f0e380b1860 2024-03-29T10:44:36.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 >> v1:172.21.15.98:6790/0 conn(0x7f0e3809ac40 legacy=0x7f0e3813ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 >> v1:172.21.15.189:6789/0 conn(0x7f0e3813b5b0 legacy=0x7f0e381e2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:36.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 monclient: found mon.a 2024-03-29T10:44:36.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:36.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:36.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e381cccd0 con 0x7f0e380b1860 2024-03-29T10:44:36.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient: _finish_auth 0 2024-03-29T10:44:36.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:36.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:36.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e3e216640 5 monclient: authenticate success, global_id 4161 2024-03-29T10:44:36.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e3e216640 10 monclient: _renew_subs 2024-03-29T10:44:36.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e3e216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:36.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e3e216640 1 -- 172.21.15.189:0/2595279121 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e381cce60 con 0x7f0e380b1860 2024-03-29T10:44:36.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e3e216640 10 client.4161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:36.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e3e216640 10 monclient: _renew_subs 2024-03-29T10:44:36.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e3e216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:36.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e3e216640 1 -- 172.21.15.189:0/2595279121 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e381d7190 con 0x7f0e380b1860 2024-03-29T10:44:36.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e3e216640 1 librados: init done 2024-03-29T10:44:36.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e28003610 con 0x7f0e380b1860 2024-03-29T10:44:36.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:36.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f0e28003920 con 0x7f0e380b1860 2024-03-29T10:44:36.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:36.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 monclient: dump: 2024-03-29T10:44:36.213 INFO:teuthology.orchestra.run.smithi189.stderr:epoch 1 2024-03-29T10:44:36.213 INFO:teuthology.orchestra.run.smithi189.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.214 INFO:teuthology.orchestra.run.smithi189.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.214 INFO:teuthology.orchestra.run.smithi189.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.214 INFO:teuthology.orchestra.run.smithi189.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.214 INFO:teuthology.orchestra.run.smithi189.stderr:election_strategy: 1 2024-03-29T10:44:36.214 INFO:teuthology.orchestra.run.smithi189.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.214 INFO:teuthology.orchestra.run.smithi189.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.214 INFO:teuthology.orchestra.run.smithi189.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.214 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:44:36.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e35ffb640 4 set_mon_vals no callback set 2024-03-29T10:44:36.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0e28003e10 con 0x7f0e380b1860 2024-03-29T10:44:36.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1420913313 0 0) 0x7f0e280056b0 con 0x7f0e380b1860 2024-03-29T10:44:36.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 10 client.4161.objecter ms_dispatch 0x7f0e38000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:44:36.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 3 client.4161.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:44:36.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 3 client.4161.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:44:36.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.209+0000 7f0e1ffff640 20 client.4161.objecter dump_active .. 0 homeless 2024-03-29T10:44:36.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.210+0000 7f0e3e216640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:36.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.210+0000 7f0e3e216640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:36.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.210+0000 7f0e3e216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:36.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.210+0000 7f0e3e216640 1 -- 172.21.15.189:0/2595279121 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e18005080 con 0x7f0e380b1860 2024-03-29T10:44:36.215 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.212+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0e280050b0 con 0x7f0e380b1860 2024-03-29T10:44:36.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.212+0000 7f0e1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:36.216 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.212+0000 7f0e1ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:36.216 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:36.216+0000 7f546422c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:44:36.267 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.284+0000 7f0e3e216640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2132347f-1364-46a0-9722-bf70172452ac", "id": 7}] 2024-03-29T10:44:36.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.284+0000 7f0e3e216640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2132347f-1364-46a0-9722-bf70172452ac", "id": 7}] 2024-03-29T10:44:36.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.284+0000 7f0e3e216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:36.268 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.284+0000 7f0e3e216640 1 -- 172.21.15.189:0/2595279121 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2132347f-1364-46a0-9722-bf70172452ac", "id": 7} v 0) v1 -- 0x7f0e180058b0 con 0x7f0e380b1860 2024-03-29T10:44:36.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.286+0000 7f0e1ffff640 1 -- 172.21.15.189:0/2595279121 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2132347f-1364-46a0-9722-bf70172452ac", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1290364117 0 1371665406) 0x7f0e28005420 con 0x7f0e380b1860 2024-03-29T10:44:36.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.286+0000 7f0e1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2132347f-1364-46a0-9722-bf70172452ac", "id": 7}] 2024-03-29T10:44:36.269 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.286+0000 7f0e1ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:36.269 INFO:teuthology.orchestra.run.smithi189.stdout:7 2024-03-29T10:44:36.270 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.287+0000 7f0e3e216640 10 librados: watch_flush enter 2024-03-29T10:44:36.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.287+0000 7f0e3e216640 10 librados: watch_flush exit 2024-03-29T10:44:36.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.287+0000 7f0e3e216640 20 client.4161.objecter shutdown clearing up homeless session... 2024-03-29T10:44:36.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.287+0000 7f0e3e216640 10 client.4161.objecter successfully canceled tick 2024-03-29T10:44:36.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.287+0000 7f0e3e216640 10 monclient: shutdown 2024-03-29T10:44:36.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.287+0000 7f0e3e216640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:36.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.287+0000 7f0e3e216640 1 -- 172.21.15.189:0/2595279121 >> v1:172.21.15.98:6789/0 conn(0x7f0e380b1860 legacy=0x7f0e3813aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.287+0000 7f0e3e216640 1 -- 172.21.15.189:0/2595279121 shutdown_connections 2024-03-29T10:44:36.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.287+0000 7f0e3e216640 1 -- 172.21.15.189:0/2595279121 >> 172.21.15.189:0/2595279121 conn(0x7f0e38066b00 msgr2=0x7f0e3813cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:36.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.287+0000 7f0e3e216640 1 -- 172.21.15.189:0/2595279121 shutdown_connections 2024-03-29T10:44:36.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.287+0000 7f0e3e216640 1 -- 172.21.15.189:0/2595279121 wait complete. 2024-03-29T10:44:36.271 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-29T10:44:36.287+0000 7f0e3e216640 1 librados: shutdown 2024-03-29T10:44:36.273 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:36.290+0000 7f3732c0f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:44:36.281 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:44:36.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:44:36.283 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:44:36.283 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:44:36.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:44:36.286 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:44:36.286 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:44:36.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:44:36.288 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:44:36.288 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:44:36.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:44:36.291 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:44:36.291 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:44:36.291 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:44:36.293 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:44:36.294 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:44:36.294 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:44:36.296 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:44:36.296 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:44:36.296 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:44:36.298 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:44:36.298 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:44:36.299 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:44:36.301 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:44:36.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:36.360 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:36.378+0000 7f546422c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:44:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.435+0000 7fd8c27ba640 10 monclient: get_monmap_and_config 2024-03-29T10:44:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.435+0000 7fd8c27ba640 10 monclient: build_initial_monmap 2024-03-29T10:44:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.435+0000 7fd8c27ba640 10 monclient: monmap: 2024-03-29T10:44:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-29T10:44:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.418 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T10:44:36.436644+0000 2024-03-29T10:44:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T10:44:36.436644+0000 2024-03-29T10:44:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:36.419 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:36.419 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:36.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.436+0000 7fd8c27ba640 1 Processor -- start 2024-03-29T10:44:36.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.436+0000 7fd8c27ba640 1 -- start start 2024-03-29T10:44:36.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.436+0000 7fd8c27ba640 10 monclient: init 2024-03-29T10:44:36.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8c27ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:36.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8c27ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:36.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8c27ba640 10 monclient(hunting): picked mon.noname-a con 0x7fd8bc0b1780 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:36.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8c27ba640 10 monclient(hunting): picked mon.noname-b con 0x7fd8bc13d7e0 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:36.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8c27ba640 10 monclient(hunting): picked mon.noname-c con 0x7fd8bc143be0 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:36.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8c27ba640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc070b30 con 0x7fd8bc0b1780 2024-03-29T10:44:36.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8c27ba640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc070da0 con 0x7fd8bc143be0 2024-03-29T10:44:36.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8c27ba640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc09adb0 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8c27ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8c27ba640 10 monclient(hunting): authenticate will time out at 896.634182s 2024-03-29T10:44:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8bbfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd8bc0b1780 0x7fd8bc13d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53984/0 (socket says 172.21.15.98:53984) 2024-03-29T10:44:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8bbfff640 1 -- 172.21.15.98:0/1688297271 learned_addr learned my addr 172.21.15.98:0/1688297271 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:44:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 1 -- 172.21.15.98:0/1688297271 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934286829 0 0) 0x7fd8bc070b30 con 0x7fd8bc0b1780 2024-03-29T10:44:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:44:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 1 -- 172.21.15.98:0/1688297271 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a4003520 con 0x7fd8bc0b1780 2024-03-29T10:44:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 1 -- 172.21.15.98:0/1688297271 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271346059 0 0) 0x7fd8bc09adb0 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 10 monclient(hunting): my global_id is 4178 2024-03-29T10:44:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 1 -- 172.21.15.98:0/1688297271 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8bc070b30 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 1 -- 172.21.15.98:0/1688297271 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167989600 0 0) 0x7fd8bc070da0 con 0x7fd8bc143be0 2024-03-29T10:44:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 10 monclient(hunting): my global_id is 4180 2024-03-29T10:44:36.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 1 -- 172.21.15.98:0/1688297271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8bc09adb0 con 0x7fd8bc143be0 2024-03-29T10:44:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 1 -- 172.21.15.98:0/1688297271 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3513576834 0 0) 0x7fd8a4003520 con 0x7fd8bc0b1780 2024-03-29T10:44:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 1 -- 172.21.15.98:0/1688297271 >> v1:172.21.15.98:6790/0 conn(0x7fd8bc13d7e0 legacy=0x7fd8bc1434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 1 -- 172.21.15.98:0/1688297271 >> v1:172.21.15.189:6789/0 conn(0x7fd8bc143be0 legacy=0x7fd8bc14b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 1 monclient: found mon.noname-a 2024-03-29T10:44:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.437+0000 7fd8baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:44:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 1 -- 172.21.15.98:0/1688297271 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc144070 con 0x7fd8bc0b1780 2024-03-29T10:44:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:36.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 1 -- 172.21.15.98:0/1688297271 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd8ac001690 con 0x7fd8bc0b1780 2024-03-29T10:44:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 10 monclient: dump: 2024-03-29T10:44:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T10:44:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.432 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.432 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 1 -- 172.21.15.98:0/1688297271 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8ac0018e0 con 0x7fd8bc0b1780 2024-03-29T10:44:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8c27ba640 5 monclient: authenticate success, global_id 4188 2024-03-29T10:44:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8c27ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 1 -- 172.21.15.98:0/1688297271 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd8ac001c70 con 0x7fd8bc0b1780 2024-03-29T10:44:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8c27ba640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8c27ba640 4 set_mon_vals no callback set 2024-03-29T10:44:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8baffd640 10 monclient: dump: 2024-03-29T10:44:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T10:44:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.434 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.434 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.434 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.434 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:36.434 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.434 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.434 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.434 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:36.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8c27ba640 10 monclient: shutdown 2024-03-29T10:44:36.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8c27ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:36.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8c27ba640 1 -- 172.21.15.98:0/1688297271 >> v1:172.21.15.98:6789/0 conn(0x7fd8bc0b1780 legacy=0x7fd8bc13d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8c27ba640 1 -- 172.21.15.98:0/1688297271 shutdown_connections 2024-03-29T10:44:36.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8c27ba640 1 -- 172.21.15.98:0/1688297271 >> 172.21.15.98:0/1688297271 conn(0x7fd8bc0b58f0 msgr2=0x7fd8bc0b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:36.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8c27ba640 1 -- 172.21.15.98:0/1688297271 shutdown_connections 2024-03-29T10:44:36.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.438+0000 7fd8c27ba640 1 -- 172.21.15.98:0/1688297271 wait complete. 2024-03-29T10:44:36.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.439+0000 7fd8c27ba640 10 monclient: build_initial_monmap 2024-03-29T10:44:36.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.439+0000 7fd8c27ba640 10 monclient: monmap: 2024-03-29T10:44:36.435 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-29T10:44:36.435 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:36.435 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-29T10:44:36.435 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-29T10:44:36.435 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:36.435 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:36.436 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.439+0000 7fd8c27ba640 1 librados: starting msgr at 2024-03-29T10:44:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.439+0000 7fd8c27ba640 1 librados: starting objecter 2024-03-29T10:44:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.439+0000 7fd8c27ba640 1 Processor -- start 2024-03-29T10:44:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 1 -- start start 2024-03-29T10:44:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 1 librados: setting wanted keys 2024-03-29T10:44:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 1 librados: calling monclient init 2024-03-29T10:44:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 10 monclient: init 2024-03-29T10:44:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 10 monclient(hunting): picked mon.noname-c con 0x7fd8bc0b1780 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 10 monclient(hunting): picked mon.noname-a con 0x7fd8bc13d7e0 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 10 monclient(hunting): picked mon.noname-b con 0x7fd8bc143be0 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc13f140 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc1d7c90 con 0x7fd8bc143be0 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc1d78b0 con 0x7fd8bc0b1780 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8c27ba640 10 monclient(hunting): authenticate will time out at 896.637659s 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8bbfff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fd8bc0b1780 0x7fd8bc1d81b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:50644/0 (socket says 172.21.15.98:50644) 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8bbfff640 1 -- 172.21.15.98:0/161491444 learned_addr learned my addr 172.21.15.98:0/161491444 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 client.?.objecter ms_handle_connect 0x7fd8bc0b1780 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 client.?.objecter ms_handle_connect 0x7fd8bc13d7e0 2024-03-29T10:44:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 client.?.objecter ms_handle_connect 0x7fd8bc143be0 2024-03-29T10:44:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270732738 0 0) 0x7fd8bc1d78b0 con 0x7fd8bc0b1780 2024-03-29T10:44:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 monclient(hunting): my global_id is 4181 2024-03-29T10:44:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a8004eb0 con 0x7fd8bc0b1780 2024-03-29T10:44:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195113362 0 0) 0x7fd8bc13f140 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.440+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8bc1d78b0 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990861212 0 0) 0x7fd8bc1d7c90 con 0x7fd8bc143be0 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): my global_id is 4183 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8bc13f140 con 0x7fd8bc143be0 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4287498157 0 0) 0x7fd8bc1d78b0 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8bc1d7c90 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd8b0001a50 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:36.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:36.440 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T10:44:36.440 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.440 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.440 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.440 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.440 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:36.440 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.440 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.441 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3855807234 0 0) 0x7fd8a8004eb0 con 0x7fd8bc0b1780 2024-03-29T10:44:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8bc1d78b0 con 0x7fd8bc0b1780 2024-03-29T10:44:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd8ac001bf0 con 0x7fd8bc0b1780 2024-03-29T10:44:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:36.441 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T10:44:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.442 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:36.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 7074235 0 0) 0x7fd8bc13f140 con 0x7fd8bc143be0 2024-03-29T10:44:36.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8a8004eb0 con 0x7fd8bc143be0 2024-03-29T10:44:36.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd8b4001a30 con 0x7fd8bc143be0 2024-03-29T10:44:36.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:36.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:36.445 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T10:44:36.446 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.446 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.446 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.447 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.447 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:36.447 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.447 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.447 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.447 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:36.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3560380212 0 0) 0x7fd8bc1d7c90 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 >> v1:172.21.15.98:6790/0 conn(0x7fd8bc0b1780 legacy=0x7fd8bc1d81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 >> v1:172.21.15.189:6789/0 conn(0x7fd8bc143be0 legacy=0x7fd8bc1359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:44:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 monclient: found mon.a 2024-03-29T10:44:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc1d7a40 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:44:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:44:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8c27ba640 5 monclient: authenticate success, global_id 4191 2024-03-29T10:44:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8c27ba640 10 monclient: _renew_subs 2024-03-29T10:44:36.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8c27ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:36.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8c27ba640 1 -- 172.21.15.98:0/161491444 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8bc140c60 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8c27ba640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:36.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8c27ba640 10 monclient: _renew_subs 2024-03-29T10:44:36.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8c27ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:36.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8c27ba640 1 -- 172.21.15.98:0/161491444 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8bc13df20 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8c27ba640 1 librados: init done 2024-03-29T10:44:36.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8b00035f0 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:36.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd8b0003900 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:36.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:36.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 monclient: dump: 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8ba7fc640 4 set_mon_vals no callback set 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd8b0003e70 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fd8b00057d0 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 10 client.4191.objecter ms_dispatch 0x7fd8bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.441+0000 7fd8b8ff9640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T10:44:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.442+0000 7fd8c27ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.442+0000 7fd8c27ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.442+0000 7fd8c27ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.442+0000 7fd8c27ba640 1 -- 172.21.15.98:0/161491444 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd89c005080 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.444+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd8b0005a60 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.444+0000 7fd8b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:36.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.444+0000 7fd8b8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:36.452 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:36.464+0000 7f3732c0f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:44:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.504+0000 7fd8c27ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.505+0000 7fd8c27ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.505+0000 7fd8c27ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:44:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.505+0000 7fd8c27ba640 1 -- 172.21.15.98:0/161491444 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd89c005330 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.505+0000 7fd8b8ff9640 1 -- 172.21.15.98:0/161491444 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fd8b00052c0 con 0x7fd8bc13d7e0 2024-03-29T10:44:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.505+0000 7fd8b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.505+0000 7fd8b8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:36.489 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-29T10:44:36.489 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.506+0000 7fd8c27ba640 10 librados: watch_flush enter 2024-03-29T10:44:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.506+0000 7fd8c27ba640 10 librados: watch_flush exit 2024-03-29T10:44:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.506+0000 7fd8c27ba640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T10:44:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.506+0000 7fd8c27ba640 10 client.4191.objecter successfully canceled tick 2024-03-29T10:44:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.506+0000 7fd8c27ba640 10 monclient: shutdown 2024-03-29T10:44:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.506+0000 7fd8c27ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.506+0000 7fd8c27ba640 1 -- 172.21.15.98:0/161491444 >> v1:172.21.15.98:6789/0 conn(0x7fd8bc13d7e0 legacy=0x7fd8bc13f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.506+0000 7fd8c27ba640 1 -- 172.21.15.98:0/161491444 shutdown_connections 2024-03-29T10:44:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.506+0000 7fd8c27ba640 1 -- 172.21.15.98:0/161491444 >> 172.21.15.98:0/161491444 conn(0x7fd8bc0b58f0 msgr2=0x7fd8bc069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.506+0000 7fd8c27ba640 1 -- 172.21.15.98:0/161491444 shutdown_connections 2024-03-29T10:44:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.506+0000 7fd8c27ba640 1 -- 172.21.15.98:0/161491444 wait complete. 2024-03-29T10:44:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-29T10:44:36.506+0000 7fd8c27ba640 1 librados: shutdown 2024-03-29T10:44:36.494 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:36.512+0000 7f3732c0f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:44:36.499 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:44:36.499 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:44:36.500 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:36.517+0000 7f546422c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:44:36.545 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:36.562+0000 7f546422c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:44:36.608 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:36.626+0000 7f3732c0f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:44:36.660 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:36.677+0000 7f546422c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:44:36.705 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:36.723+0000 7f3732c0f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:44:36.756 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:36.773+0000 7f546422c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:44:36.860 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:36.877+0000 7f3732c0f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:44:36.880 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-03-29T10:44:36.897+0000 7fe886e10640 -1 Falling back to public interface 2024-03-29T10:44:36.893 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-03-29T10:44:36.910+0000 7f0c4b82a640 -1 Falling back to public interface 2024-03-29T10:44:36.894 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-03-29T10:44:36.910+0000 7f38b6381640 -1 Falling back to public interface 2024-03-29T10:44:36.894 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-03-29T10:44:36.910+0000 7f38b5330640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6800/0: (98) Address already in use 2024-03-29T10:44:36.902 INFO:tasks.ceph.osd.7.smithi189.stderr:2024-03-29T10:44:36.919+0000 7f305fe0f640 -1 Falling back to public interface 2024-03-29T10:44:36.903 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:36.921+0000 7f3732c0f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:44:36.909 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-03-29T10:44:36.926+0000 7f42d0545640 -1 Falling back to public interface 2024-03-29T10:44:36.910 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:36.927+0000 7f546422c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:44:36.913 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-03-29T10:44:36.930+0000 7f41018f7640 -1 Falling back to public interface 2024-03-29T10:44:36.913 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-03-29T10:44:36.930+0000 7fc864e24640 -1 Falling back to public interface 2024-03-29T10:44:36.913 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-03-29T10:44:36.930+0000 7fc863dd3640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6808/0: (98) Address already in use 2024-03-29T10:44:36.923 INFO:tasks.ceph.osd.6.smithi189.stderr:2024-03-29T10:44:36.941+0000 7ff8c2c1c640 -1 Falling back to public interface 2024-03-29T10:44:36.957 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:36.974+0000 7f546422c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:44:37.057 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:37.074+0000 7f3732c0f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:44:37.108 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:37.125+0000 7f546422c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:44:37.305 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:37.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.411+0000 7f9567a84640 10 monclient: get_monmap_and_config 2024-03-29T10:44:37.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.411+0000 7f9567a84640 10 monclient: build_initial_monmap 2024-03-29T10:44:37.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.411+0000 7f9567a84640 10 monclient: monmap: 2024-03-29T10:44:37.394 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:44:37.394 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:37.394 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:44:37.412008+0000 2024-03-29T10:44:37.394 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:44:37.412008+0000 2024-03-29T10:44:37.394 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:37.394 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:37.394 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:37.394 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:37.395 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:37.395 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:37.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.411+0000 7f9567a84640 1 Processor -- start 2024-03-29T10:44:37.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.411+0000 7f9567a84640 1 -- start start 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.411+0000 7f9567a84640 10 monclient: init 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f9567a84640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f9567a84640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f9567a84640 10 monclient(hunting): picked mon.noname-a con 0x7f95601422b0 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f9567a84640 10 monclient(hunting): picked mon.noname-c con 0x7f9560142690 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f9567a84640 10 monclient(hunting): picked mon.noname-b con 0x7f95600b17d0 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f9567a84640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560142c80 con 0x7f95601422b0 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f9567a84640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95600aae40 con 0x7f9560142690 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f9567a84640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956013fbd0 con 0x7f95600b17d0 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f9567a84640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f9567a84640 10 monclient(hunting): authenticate will time out at 897.609594s 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f9565820640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f95601422b0 0x7f956013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53992/0 (socket says 172.21.15.98:53992) 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f9565820640 1 -- 172.21.15.98:0/3469328173 learned_addr learned my addr 172.21.15.98:0/3469328173 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:44:37.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f9566021640 1 --1- 172.21.15.98:0/3469328173 >> v1:172.21.15.98:6790/0 conn(0x7f95600b17d0 0x7f95600b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:50658/0 (socket says 172.21.15.98:50658) 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f956481e640 1 -- 172.21.15.98:0/3469328173 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917724681 0 0) 0x7f9560142c80 con 0x7f95601422b0 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f956481e640 10 monclient(hunting): _negotiate 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f956481e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.412+0000 7f956481e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient(hunting): my global_id is 4194 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 1 -- 172.21.15.98:0/3469328173 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f954c003520 con 0x7f95601422b0 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 1 -- 172.21.15.98:0/3469328173 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859714725 0 0) 0x7f956013fbd0 con 0x7f95600b17d0 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient(hunting): _negotiate 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient(hunting): my global_id is 4184 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 1 -- 172.21.15.98:0/3469328173 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9560142c80 con 0x7f95600b17d0 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 1 -- 172.21.15.98:0/3469328173 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347688727 0 0) 0x7f95600aae40 con 0x7f9560142690 2024-03-29T10:44:37.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient(hunting): _negotiate 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient(hunting): my global_id is 4186 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 1 -- 172.21.15.98:0/3469328173 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f956013fbd0 con 0x7f9560142690 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 1 -- 172.21.15.98:0/3469328173 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2347442765 0 0) 0x7f9560142c80 con 0x7f95600b17d0 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 1 -- 172.21.15.98:0/3469328173 >> v1:172.21.15.189:6789/0 conn(0x7f9560142690 legacy=0x7f95600b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 1 -- 172.21.15.98:0/3469328173 >> v1:172.21.15.98:6789/0 conn(0x7f95601422b0 legacy=0x7f956013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient: _finish_hunting 0 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 1 monclient: found mon.noname-b 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 1 -- 172.21.15.98:0/3469328173 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9560151940 con 0x7f95600b17d0 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient: _finish_auth 0 2024-03-29T10:44:37.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient: _check_auth_tickets 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f9567a84640 5 monclient: authenticate success, global_id 4184 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 1 -- 172.21.15.98:0/3469328173 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f95580018a0 con 0x7f95600b17d0 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f9567a84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient: dump: 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:37.407 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f9567a84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 1 -- 172.21.15.98:0/3469328173 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9558001a00 con 0x7f95600b17d0 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f9567a84640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f9567a84640 4 set_mon_vals no callback set 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 1 -- 172.21.15.98:0/3469328173 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f9558001dd0 con 0x7f95600b17d0 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f956481e640 10 monclient: dump: 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:37.408 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:37.409 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:37.409 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:37.409 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:37.409 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:37.409 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:37.409 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:37.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f9567a84640 10 monclient: shutdown 2024-03-29T10:44:37.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f9567a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:37.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f9567a84640 1 -- 172.21.15.98:0/3469328173 >> v1:172.21.15.98:6790/0 conn(0x7f95600b17d0 legacy=0x7f95600b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:37.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f9567a84640 1 -- 172.21.15.98:0/3469328173 shutdown_connections 2024-03-29T10:44:37.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.413+0000 7f9567a84640 1 -- 172.21.15.98:0/3469328173 >> 172.21.15.98:0/3469328173 conn(0x7f9560066a00 msgr2=0x7f9560066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:37.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.414+0000 7f9567a84640 1 -- 172.21.15.98:0/3469328173 shutdown_connections 2024-03-29T10:44:37.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.414+0000 7f9567a84640 1 -- 172.21.15.98:0/3469328173 wait complete. 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.414+0000 7f9567a84640 10 monclient: build_initial_monmap 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.414+0000 7f9567a84640 10 monclient: monmap: 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.414+0000 7f9567a84640 1 librados: starting msgr at 2024-03-29T10:44:37.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.414+0000 7f9567a84640 1 librados: starting objecter 2024-03-29T10:44:37.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.414+0000 7f9567a84640 1 Processor -- start 2024-03-29T10:44:37.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 1 -- start start 2024-03-29T10:44:37.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 1 librados: setting wanted keys 2024-03-29T10:44:37.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 1 librados: calling monclient init 2024-03-29T10:44:37.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 10 monclient: init 2024-03-29T10:44:37.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:37.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:37.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 10 monclient(hunting): picked mon.noname-b con 0x7f95600b17d0 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:37.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 10 monclient(hunting): picked mon.noname-a con 0x7f95601422b0 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:37.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 10 monclient(hunting): picked mon.noname-c con 0x7f9560142690 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:37.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95600ab670 con 0x7f95601422b0 2024-03-29T10:44:37.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95601dafa0 con 0x7f95600b17d0 2024-03-29T10:44:37.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956014e250 con 0x7f9560142690 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9567a84640 10 monclient(hunting): authenticate will time out at 897.612490s 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9566021640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f9560142690 0x7f956014db40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:50670/0 (socket says 172.21.15.98:50670) 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f9566021640 1 -- 172.21.15.98:0/1488539049 learned_addr learned my addr 172.21.15.98:0/1488539049 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f956501f640 1 --1- 172.21.15.98:0/1488539049 >> v1:172.21.15.98:6789/0 conn(0x7f95601422b0 0x7f956014d430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54004/0 (socket says 172.21.15.98:54004) 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 client.?.objecter ms_handle_connect 0x7f9560142690 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 client.?.objecter ms_handle_connect 0x7f95601422b0 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 client.?.objecter ms_handle_connect 0x7f95600b17d0 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938307808 0 0) 0x7f956014e250 con 0x7f9560142690 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:37.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 monclient(hunting): my global_id is 4187 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9544004eb0 con 0x7f9560142690 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653227207 0 0) 0x7f95600ab670 con 0x7f95601422b0 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 monclient(hunting): my global_id is 4197 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f956014e250 con 0x7f95601422b0 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497259467 0 0) 0x7f95601dafa0 con 0x7f95600b17d0 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:37.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 10 monclient(hunting): my global_id is 4189 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.415+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95600ab670 con 0x7f95600b17d0 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 594975893 0 0) 0x7f9544004eb0 con 0x7f9560142690 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95601dafa0 con 0x7f9560142690 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f9558001860 con 0x7f9560142690 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient(hunting): dump: 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:37.414 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2052146479 0 0) 0x7f95600ab670 con 0x7f95600b17d0 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9544004eb0 con 0x7f95600b17d0 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3699147393 0 0) 0x7f956014e250 con 0x7f95601422b0 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95600ab670 con 0x7f95601422b0 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f955c001230 con 0x7f95600b17d0 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient(hunting): dump: 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:37.415 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f95500017d0 con 0x7f95601422b0 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient(hunting): dump: 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:37.416 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 334162668 0 0) 0x7f9544004eb0 con 0x7f95600b17d0 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 >> v1:172.21.15.98:6790/0 conn(0x7f9560142690 legacy=0x7f956014db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 >> v1:172.21.15.98:6789/0 conn(0x7f95601422b0 legacy=0x7f956014d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 monclient: found mon.b 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:37.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95600acc80 con 0x7f95600b17d0 2024-03-29T10:44:37.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:37.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:37.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:37.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f9567a84640 5 monclient: authenticate success, global_id 4189 2024-03-29T10:44:37.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f9567a84640 10 monclient: _renew_subs 2024-03-29T10:44:37.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f9567a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:37.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f9567a84640 1 -- 172.21.15.98:0/1488539049 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95600acfc0 con 0x7f95600b17d0 2024-03-29T10:44:37.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f9567a84640 10 client.4189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:37.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f9567a84640 10 monclient: _renew_subs 2024-03-29T10:44:37.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f9567a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:37.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f9567a84640 1 -- 172.21.15.98:0/1488539049 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9560151110 con 0x7f95600b17d0 2024-03-29T10:44:37.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f9567a84640 1 librados: init done 2024-03-29T10:44:37.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f955c001cb0 con 0x7f95600b17d0 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f955c0047e0 con 0x7f95600b17d0 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 monclient: dump: 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:37.419 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f9557fff640 4 set_mon_vals no callback set 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f955c004d90 con 0x7f95600b17d0 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f955c0066a0 con 0x7f95600b17d0 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 10 client.4189.objecter ms_dispatch 0x7f9560000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 3 client.4189.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 3 client.4189.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.416+0000 7f95567fc640 20 client.4189.objecter dump_active .. 0 homeless 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.417+0000 7f9567a84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.417+0000 7f9567a84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.417+0000 7f9567a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.417+0000 7f9567a84640 1 -- 172.21.15.98:0/1488539049 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95600b2720 con 0x7f95600b17d0 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.419+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f955c006080 con 0x7f95600b17d0 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.419+0000 7f95567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:37.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.419+0000 7f95567fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:37.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.481+0000 7f9567a84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:37.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.481+0000 7f9567a84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:37.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.481+0000 7f9567a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:37.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.481+0000 7f9567a84640 1 -- 172.21.15.98:0/1488539049 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9560075f10 con 0x7f95600b17d0 2024-03-29T10:44:37.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f95567fc640 1 -- 172.21.15.98:0/1488539049 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f955c0063d0 con 0x7f95600b17d0 2024-03-29T10:44:37.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f95567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:37.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f95567fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:37.465 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:44:37.465 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:37.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f9567a84640 10 librados: watch_flush enter 2024-03-29T10:44:37.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f9567a84640 10 librados: watch_flush exit 2024-03-29T10:44:37.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f9567a84640 20 client.4189.objecter shutdown clearing up homeless session... 2024-03-29T10:44:37.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f9567a84640 10 client.4189.objecter successfully canceled tick 2024-03-29T10:44:37.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f9567a84640 10 monclient: shutdown 2024-03-29T10:44:37.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f9567a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:37.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f9567a84640 1 -- 172.21.15.98:0/1488539049 >> v1:172.21.15.189:6789/0 conn(0x7f95600b17d0 legacy=0x7f95601da890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:37.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f9567a84640 1 -- 172.21.15.98:0/1488539049 shutdown_connections 2024-03-29T10:44:37.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f9567a84640 1 -- 172.21.15.98:0/1488539049 >> 172.21.15.98:0/1488539049 conn(0x7f9560066a00 msgr2=0x7f956013f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:37.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f9567a84640 1 -- 172.21.15.98:0/1488539049 shutdown_connections 2024-03-29T10:44:37.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.482+0000 7f9567a84640 1 -- 172.21.15.98:0/1488539049 wait complete. 2024-03-29T10:44:37.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:37.483+0000 7f9567a84640 1 librados: shutdown 2024-03-29T10:44:37.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:37.873 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:37.891+0000 7f3732c0f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:44:37.947 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:37.964+0000 7f546422c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:44:38.147 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:38.164+0000 7f3732c0f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:44:38.198 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:38.215+0000 7f3732c0f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:44:38.222 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:38.240+0000 7f546422c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:44:38.275 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:38.293+0000 7f546422c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:44:38.315 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:38.333+0000 7f3732c0f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:44:38.394 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-03-29T10:44:38.412+0000 7f546422c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:44:39.331 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:39.349+0000 7f36d3455640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:44:39.333 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-03-29T10:44:39.350+0000 7f36d3455640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:44:40.350 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-03-29T10:44:40.368+0000 7fe886e10640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:44:40.402 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-03-29T10:44:40.420+0000 7f38b6381640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:44:40.425 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-03-29T10:44:40.443+0000 7fc864e24640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:44:40.427 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-03-29T10:44:40.444+0000 7f41018f7640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:44:40.431 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-03-29T10:44:40.449+0000 7f42d0545640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:44:40.435 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-03-29T10:44:40.452+0000 7f0c4b82a640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:44:40.442 INFO:tasks.ceph.osd.7.smithi189.stderr:2024-03-29T10:44:40.460+0000 7f305fe0f640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:44:40.470 INFO:tasks.ceph.osd.6.smithi189.stderr:2024-03-29T10:44:40.487+0000 7ff8c2c1c640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:44:44.281 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:44.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.459+0000 7f4e4e36c640 10 monclient: get_monmap_and_config 2024-03-29T10:44:44.443 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.459+0000 7f4e4e36c640 10 monclient: build_initial_monmap 2024-03-29T10:44:44.443 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.459+0000 7f4e4e36c640 10 monclient: monmap: 2024-03-29T10:44:44.443 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:44:44.443 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:44.443 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:44:44.460577+0000 2024-03-29T10:44:44.443 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:44:44.460577+0000 2024-03-29T10:44:44.443 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:44.443 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:44.443 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:44.443 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:44.443 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:44.443 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:44.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.461+0000 7f4e4e36c640 1 Processor -- start 2024-03-29T10:44:44.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.461+0000 7f4e4e36c640 1 -- start start 2024-03-29T10:44:44.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.461+0000 7f4e4e36c640 10 monclient: init 2024-03-29T10:44:44.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.462+0000 7f4e4e36c640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:44.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.462+0000 7f4e4e36c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:44.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.462+0000 7f4e4e36c640 10 monclient(hunting): picked mon.noname-b con 0x7f4e480b0770 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:44.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.462+0000 7f4e4e36c640 10 monclient(hunting): picked mon.noname-a con 0x7f4e4814a700 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:44.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.462+0000 7f4e4e36c640 10 monclient(hunting): picked mon.noname-c con 0x7f4e480ac5a0 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:44.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.462+0000 7f4e4e36c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e480aba70 con 0x7f4e4814a700 2024-03-29T10:44:44.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.462+0000 7f4e4e36c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e480aecb0 con 0x7f4e480ac5a0 2024-03-29T10:44:44.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.462+0000 7f4e4c909640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4e480ac5a0 0x7f4e48151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:45204/0 (socket says 172.21.15.98:45204) 2024-03-29T10:44:44.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.462+0000 7f4e4c909640 1 -- 172.21.15.98:0/245478270 learned_addr learned my addr 172.21.15.98:0/245478270 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:44:44.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.462+0000 7f4e4e36c640 1 -- 172.21.15.98:0/245478270 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e480af150 con 0x7f4e480b0770 2024-03-29T10:44:44.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.462+0000 7f4e4e36c640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:44.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.462+0000 7f4e4e36c640 10 monclient(hunting): authenticate will time out at 904.659573s 2024-03-29T10:44:44.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 1 -- 172.21.15.98:0/245478270 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383265144 0 0) 0x7f4e480aba70 con 0x7f4e4814a700 2024-03-29T10:44:44.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:44.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:44.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:44.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient(hunting): my global_id is 4239 2024-03-29T10:44:44.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 1 -- 172.21.15.98:0/245478270 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e30003520 con 0x7f4e4814a700 2024-03-29T10:44:44.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 1 -- 172.21.15.98:0/245478270 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688318879 0 0) 0x7f4e480aecb0 con 0x7f4e480ac5a0 2024-03-29T10:44:44.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:44.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:44.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:44.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient(hunting): my global_id is 4231 2024-03-29T10:44:44.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 1 -- 172.21.15.98:0/245478270 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e480aba70 con 0x7f4e480ac5a0 2024-03-29T10:44:44.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 1 -- 172.21.15.98:0/245478270 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3220657864 0 0) 0x7f4e30003520 con 0x7f4e4814a700 2024-03-29T10:44:44.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 1 -- 172.21.15.98:0/245478270 >> v1:172.21.15.98:6790/0 conn(0x7f4e480b0770 legacy=0x7f4e48149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:44.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 1 -- 172.21.15.98:0/245478270 >> v1:172.21.15.189:6789/0 conn(0x7f4e480ac5a0 legacy=0x7f4e48151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:44.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:44.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 1 monclient: found mon.noname-a 2024-03-29T10:44:44.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:44.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:44:44.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 1 -- 172.21.15.98:0/245478270 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e480ac9d0 con 0x7f4e4814a700 2024-03-29T10:44:44.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:44.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:44.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:44.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 1 -- 172.21.15.98:0/245478270 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4e3c001690 con 0x7f4e4814a700 2024-03-29T10:44:44.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:44.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:44.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.463+0000 7f4e46ffd640 10 monclient: dump: 2024-03-29T10:44:44.452 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:44.452 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:44.452 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:44.452 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:44.452 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:44.453 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:44.453 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:44.453 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:44.453 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:44.453 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:44.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.464+0000 7f4e4e36c640 5 monclient: authenticate success, global_id 4239 2024-03-29T10:44:44.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.464+0000 7f4e4e36c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:44.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.464+0000 7f4e46ffd640 1 -- 172.21.15.98:0/245478270 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e3c0017f0 con 0x7f4e4814a700 2024-03-29T10:44:44.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.464+0000 7f4e46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:44.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.464+0000 7f4e46ffd640 1 -- 172.21.15.98:0/245478270 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4e3c001c00 con 0x7f4e4814a700 2024-03-29T10:44:44.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.464+0000 7f4e4e36c640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:44.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.464+0000 7f4e4e36c640 4 set_mon_vals no callback set 2024-03-29T10:44:44.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.464+0000 7f4e46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:44.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.464+0000 7f4e46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:44.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.464+0000 7f4e46ffd640 10 monclient: dump: 2024-03-29T10:44:44.454 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:44.455 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:44.455 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:44.455 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:44.455 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:44.455 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:44.455 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:44.455 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:44.455 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:44.455 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:44.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.465+0000 7f4e4e36c640 10 monclient: shutdown 2024-03-29T10:44:44.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.465+0000 7f4e4e36c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:44.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.465+0000 7f4e4e36c640 1 -- 172.21.15.98:0/245478270 >> v1:172.21.15.98:6789/0 conn(0x7f4e4814a700 legacy=0x7f4e480abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:44.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.465+0000 7f4e4e36c640 1 -- 172.21.15.98:0/245478270 shutdown_connections 2024-03-29T10:44:44.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.465+0000 7f4e4e36c640 1 -- 172.21.15.98:0/245478270 >> 172.21.15.98:0/245478270 conn(0x7f4e4806b660 msgr2=0x7f4e4806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:44.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.465+0000 7f4e4e36c640 1 -- 172.21.15.98:0/245478270 shutdown_connections 2024-03-29T10:44:44.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.465+0000 7f4e4e36c640 1 -- 172.21.15.98:0/245478270 wait complete. 2024-03-29T10:44:44.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.466+0000 7f4e4e36c640 10 monclient: build_initial_monmap 2024-03-29T10:44:44.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.466+0000 7f4e4e36c640 10 monclient: monmap: 2024-03-29T10:44:44.465 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:44:44.465 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:44.465 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:44:44.465 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:44:44.465 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:44.465 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:44.466 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:44.466 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:44.466 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:44.466 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:44.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.467+0000 7f4e4e36c640 1 librados: starting msgr at 2024-03-29T10:44:44.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.467+0000 7f4e4e36c640 1 librados: starting objecter 2024-03-29T10:44:44.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.467+0000 7f4e4e36c640 1 Processor -- start 2024-03-29T10:44:44.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.467+0000 7f4e4e36c640 1 -- start start 2024-03-29T10:44:44.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.467+0000 7f4e4e36c640 1 librados: setting wanted keys 2024-03-29T10:44:44.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.467+0000 7f4e4e36c640 1 librados: calling monclient init 2024-03-29T10:44:44.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.467+0000 7f4e4e36c640 10 monclient: init 2024-03-29T10:44:44.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.468+0000 7f4e4e36c640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:44.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.468+0000 7f4e4e36c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.468+0000 7f4e4e36c640 10 monclient(hunting): picked mon.noname-b con 0x7f4e480ac5a0 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.469+0000 7f4e4e36c640 10 monclient(hunting): picked mon.noname-a con 0x7f4e480b0770 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.469+0000 7f4e4e36c640 10 monclient(hunting): picked mon.noname-c con 0x7f4e4814a700 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.469+0000 7f4e4e36c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e481deb30 con 0x7f4e480b0770 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.469+0000 7f4e477fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4e480b0770 0x7f4e481df2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33448/0 (socket says 172.21.15.98:33448) 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.469+0000 7f4e4e36c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e48128330 con 0x7f4e480ac5a0 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.469+0000 7f4e477fe640 1 -- 172.21.15.98:0/3306317445 learned_addr learned my addr 172.21.15.98:0/3306317445 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.469+0000 7f4e4e36c640 1 -- 172.21.15.98:0/3306317445 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e481df9b0 con 0x7f4e4814a700 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.469+0000 7f4e4e36c640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.469+0000 7f4e4e36c640 10 monclient(hunting): authenticate will time out at 904.666338s 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.469+0000 7f4e44ff9640 10 client.?.objecter ms_handle_connect 0x7f4e480ac5a0 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.469+0000 7f4e44ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.469+0000 7f4e44ff9640 10 client.?.objecter ms_handle_connect 0x7f4e480b0770 2024-03-29T10:44:44.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.469+0000 7f4e44ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:44.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178093653 0 0) 0x7f4e48128330 con 0x7f4e480ac5a0 2024-03-29T10:44:44.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:44.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:44.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:44.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 monclient(hunting): my global_id is 4234 2024-03-29T10:44:44.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e34004eb0 con 0x7f4e480ac5a0 2024-03-29T10:44:44.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 client.?.objecter ms_handle_connect 0x7f4e4814a700 2024-03-29T10:44:44.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:44.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482675925 0 0) 0x7f4e481deb30 con 0x7f4e480b0770 2024-03-29T10:44:44.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:44.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:44.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:44.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e48128330 con 0x7f4e480b0770 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1401959587 0 0) 0x7f4e34004eb0 con 0x7f4e480ac5a0 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e481deb30 con 0x7f4e480ac5a0 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4e380018c0 con 0x7f4e480ac5a0 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:44.469 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.470+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112002940 0 0) 0x7f4e481df9b0 con 0x7f4e4814a700 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 10 monclient(hunting): my global_id is 4232 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e34005840 con 0x7f4e4814a700 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 376118452 0 0) 0x7f4e48128330 con 0x7f4e480b0770 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e481df9b0 con 0x7f4e480b0770 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4e3c001bc0 con 0x7f4e480b0770 2024-03-29T10:44:44.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2793463988 0 0) 0x7f4e481deb30 con 0x7f4e480ac5a0 2024-03-29T10:44:44.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 >> v1:172.21.15.98:6790/0 conn(0x7f4e4814a700 legacy=0x7f4e48144ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 >> v1:172.21.15.98:6789/0 conn(0x7f4e480b0770 legacy=0x7f4e481df2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 1 monclient: found mon.b 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e481ded80 con 0x7f4e480ac5a0 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 10 monclient: _finish_auth 0 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e4e36c640 5 monclient: authenticate success, global_id 4234 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e4e36c640 10 monclient: _renew_subs 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e4e36c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e4e36c640 1 -- 172.21.15.98:0/3306317445 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e481239d0 con 0x7f4e480ac5a0 2024-03-29T10:44:44.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e4e36c640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e4e36c640 10 monclient: _renew_subs 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e4e36c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e4e36c640 1 -- 172.21.15.98:0/3306317445 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e48145600 con 0x7f4e480ac5a0 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.471+0000 7f4e4e36c640 1 librados: init done 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.472+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e380035f0 con 0x7f4e480ac5a0 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.472+0000 7f4e44ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.472+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4e38003900 con 0x7f4e480ac5a0 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.472+0000 7f4e44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.472+0000 7f4e44ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.472+0000 7f4e44ff9640 10 monclient: dump: 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:44.473 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.472+0000 7f4e45ffb640 4 set_mon_vals no callback set 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.473+0000 7f4e4e36c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.473+0000 7f4e4e36c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.473+0000 7f4e4e36c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.474+0000 7f4e4e36c640 1 -- 172.21.15.98:0/3306317445 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e28005080 con 0x7f4e480ac5a0 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.475+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f4e38019950 con 0x7f4e480ac5a0 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.476+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f4e380835f0 con 0x7f4e480ac5a0 2024-03-29T10:44:44.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.476+0000 7f4e44ff9640 10 client.4234.objecter ms_dispatch 0x7f4e48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:44.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.476+0000 7f4e44ff9640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:44.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.476+0000 7f4e44ff9640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:44.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.476+0000 7f4e44ff9640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-29T10:44:44.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.476+0000 7f4e477fe640 10 monclient: get_auth_request con 0x7f4e3406e310 auth_method 0 2024-03-29T10:44:44.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.476+0000 7f4e44ff9640 10 client.4234.objecter ms_handle_connect 0x7f4e3406e310 2024-03-29T10:44:44.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.483+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e3804ee40 con 0x7f4e480ac5a0 2024-03-29T10:44:44.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.483+0000 7f4e44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:44.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.483+0000 7f4e44ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:44.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.882+0000 7f4e4e36c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:44.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.882+0000 7f4e4e36c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:44.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.882+0000 7f4e4e36c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:44.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.882+0000 7f4e4e36c640 1 -- 172.21.15.98:0/3306317445 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e28005330 con 0x7f4e480ac5a0 2024-03-29T10:44:44.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.883+0000 7f4e44ff9640 1 -- 172.21.15.98:0/3306317445 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f4e38003760 con 0x7f4e480ac5a0 2024-03-29T10:44:44.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.883+0000 7f4e44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:44.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.883+0000 7f4e44ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:44.866 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:44:44.866 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:44.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.887+0000 7f4e4e36c640 10 librados: watch_flush enter 2024-03-29T10:44:44.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.887+0000 7f4e4e36c640 10 librados: watch_flush exit 2024-03-29T10:44:44.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.887+0000 7f4e4e36c640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-29T10:44:44.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.887+0000 7f4e4e36c640 10 client.4234.objecter successfully canceled tick 2024-03-29T10:44:44.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.887+0000 7f4e4e36c640 1 -- 172.21.15.98:0/3306317445 >> v1:172.21.15.98:6816/1481799426 conn(0x7f4e3406e310 legacy=0x7f4e340707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:44.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.887+0000 7f4e4e36c640 10 monclient: shutdown 2024-03-29T10:44:44.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.887+0000 7f4e4e36c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:44.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.887+0000 7f4e4e36c640 1 -- 172.21.15.98:0/3306317445 >> v1:172.21.15.189:6789/0 conn(0x7f4e480ac5a0 legacy=0x7f4e48125f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:44.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.888+0000 7f4e4e36c640 1 -- 172.21.15.98:0/3306317445 shutdown_connections 2024-03-29T10:44:44.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.888+0000 7f4e4e36c640 1 -- 172.21.15.98:0/3306317445 >> 172.21.15.98:0/3306317445 conn(0x7f4e4806b660 msgr2=0x7f4e480abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:44.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.888+0000 7f4e4e36c640 1 -- 172.21.15.98:0/3306317445 shutdown_connections 2024-03-29T10:44:44.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.888+0000 7f4e4e36c640 1 -- 172.21.15.98:0/3306317445 wait complete. 2024-03-29T10:44:44.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:44.888+0000 7f4e4e36c640 1 librados: shutdown 2024-03-29T10:44:44.887 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:51.695 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:51.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.859+0000 7f6e176d9640 10 monclient: get_monmap_and_config 2024-03-29T10:44:51.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.859+0000 7f6e176d9640 10 monclient: build_initial_monmap 2024-03-29T10:44:51.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.859+0000 7f6e176d9640 10 monclient: monmap: 2024-03-29T10:44:51.843 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:44:51.843 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:51.843 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:44:51.861428+0000 2024-03-29T10:44:51.843 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:44:51.861428+0000 2024-03-29T10:44:51.844 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:51.844 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:51.844 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:51.844 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:51.844 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:51.844 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:51.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.860+0000 7f6e176d9640 1 Processor -- start 2024-03-29T10:44:51.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.860+0000 7f6e176d9640 1 -- start start 2024-03-29T10:44:51.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.860+0000 7f6e176d9640 10 monclient: init 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.861+0000 7f6e176d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.861+0000 7f6e176d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.861+0000 7f6e176d9640 10 monclient(hunting): picked mon.noname-c con 0x7f6e100b0860 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e176d9640 10 monclient(hunting): picked mon.noname-b con 0x7f6e1013ef30 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e176d9640 10 monclient(hunting): picked mon.noname-a con 0x7f6e100af8e0 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e176d9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e100b0c40 con 0x7f6e100af8e0 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e176d9640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e100aed10 con 0x7f6e100b0860 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e176d9640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e100afd50 con 0x7f6e1013ef30 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e176d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e176d9640 10 monclient(hunting): authenticate will time out at 912.060080s 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e15475640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6e100b0860 0x7f6e1013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:35650/0 (socket says 172.21.15.98:35650) 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e15475640 1 -- 172.21.15.98:0/3343338922 learned_addr learned my addr 172.21.15.98:0/3343338922 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e07fff640 1 -- 172.21.15.98:0/3343338922 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251079354 0 0) 0x7f6e100afd50 con 0x7f6e1013ef30 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e07fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e07fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e07fff640 10 monclient(hunting): my global_id is 4235 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.862+0000 7f6e07fff640 1 -- 172.21.15.98:0/3343338922 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dfc003520 con 0x7f6e1013ef30 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 1 -- 172.21.15.98:0/3343338922 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491837197 0 0) 0x7f6e100aed10 con 0x7f6e100b0860 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 10 monclient(hunting): my global_id is 4237 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 1 -- 172.21.15.98:0/3343338922 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e100afd50 con 0x7f6e100b0860 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 1 -- 172.21.15.98:0/3343338922 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3061893575 0 0) 0x7f6dfc003520 con 0x7f6e1013ef30 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 1 -- 172.21.15.98:0/3343338922 >> v1:172.21.15.189:6789/0 conn(0x7f6e100b0860 legacy=0x7f6e1013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 1 -- 172.21.15.98:0/3343338922 >> v1:172.21.15.98:6789/0 conn(0x7f6e100af8e0 legacy=0x7f6e10149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 1 monclient: found mon.noname-b 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 1 -- 172.21.15.98:0/3343338922 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e100ab1b0 con 0x7f6e1013ef30 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 10 monclient: _finish_auth 0 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e176d9640 5 monclient: authenticate success, global_id 4235 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e176d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 1 -- 172.21.15.98:0/3343338922 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f6e00001690 con 0x7f6e1013ef30 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:51.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 10 monclient: dump: 2024-03-29T10:44:51.850 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:51.850 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:51.850 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:51.850 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:51.850 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:51.850 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:51.850 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:51.850 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:51.850 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:51.850 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:51.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e176d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:51.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 1 -- 172.21.15.98:0/3343338922 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e00001870 con 0x7f6e1013ef30 2024-03-29T10:44:51.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e176d9640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.863+0000 7f6e176d9640 4 set_mon_vals no callback set 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.864+0000 7f6e07fff640 1 -- 172.21.15.98:0/3343338922 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f6e00001690 con 0x7f6e1013ef30 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.864+0000 7f6e07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.864+0000 7f6e07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.864+0000 7f6e07fff640 10 monclient: dump: 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:51.861 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.864+0000 7f6e176d9640 10 monclient: shutdown 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.864+0000 7f6e176d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.864+0000 7f6e176d9640 1 -- 172.21.15.98:0/3343338922 >> v1:172.21.15.98:6790/0 conn(0x7f6e1013ef30 legacy=0x7f6e100af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.864+0000 7f6e176d9640 1 -- 172.21.15.98:0/3343338922 shutdown_connections 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.864+0000 7f6e176d9640 1 -- 172.21.15.98:0/3343338922 >> 172.21.15.98:0/3343338922 conn(0x7f6e1006b660 msgr2=0x7f6e1006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.864+0000 7f6e176d9640 1 -- 172.21.15.98:0/3343338922 shutdown_connections 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.865+0000 7f6e176d9640 1 -- 172.21.15.98:0/3343338922 wait complete. 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.865+0000 7f6e176d9640 10 monclient: build_initial_monmap 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.865+0000 7f6e176d9640 10 monclient: monmap: 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:44:51.862 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.866+0000 7f6e176d9640 1 librados: starting msgr at 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.866+0000 7f6e176d9640 1 librados: starting objecter 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.866+0000 7f6e176d9640 1 Processor -- start 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.867+0000 7f6e176d9640 1 -- start start 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.867+0000 7f6e176d9640 1 librados: setting wanted keys 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.867+0000 7f6e176d9640 1 librados: calling monclient init 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.867+0000 7f6e176d9640 10 monclient: init 2024-03-29T10:44:51.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.868+0000 7f6e176d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.868+0000 7f6e176d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.868+0000 7f6e176d9640 10 monclient(hunting): picked mon.noname-a con 0x7f6e100af8e0 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.868+0000 7f6e176d9640 10 monclient(hunting): picked mon.noname-c con 0x7f6e100b0860 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.868+0000 7f6e176d9640 10 monclient(hunting): picked mon.noname-b con 0x7f6e1013ef30 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.868+0000 7f6e176d9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e100b1400 con 0x7f6e100af8e0 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.868+0000 7f6e14c74640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f6e100b0860 0x7f6e101d3110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:33850/0 (socket says 172.21.15.98:33850) 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.868+0000 7f6e14c74640 1 -- 172.21.15.98:0/3453622292 learned_addr learned my addr 172.21.15.98:0/3453622292 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.868+0000 7f6e05ffb640 10 client.?.objecter ms_handle_connect 0x7f6e100b0860 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.868+0000 7f6e05ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.868+0000 7f6e176d9640 1 -- 172.21.15.98:0/3453622292 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e100b2830 con 0x7f6e1013ef30 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e176d9640 1 -- 172.21.15.98:0/3453622292 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e100b2b10 con 0x7f6e100b0860 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 client.?.objecter ms_handle_connect 0x7f6e100af8e0 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e176d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e176d9640 10 monclient(hunting): authenticate will time out at 912.067152s 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 client.?.objecter ms_handle_connect 0x7f6e1013ef30 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354599054 0 0) 0x7f6e100b1400 con 0x7f6e100af8e0 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 monclient(hunting): my global_id is 4248 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df4004eb0 con 0x7f6e100af8e0 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841208957 0 0) 0x7f6e100b2830 con 0x7f6e1013ef30 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e100b1400 con 0x7f6e1013ef30 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771258949 0 0) 0x7f6e100b2b10 con 0x7f6e100b0860 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 10 monclient(hunting): my global_id is 4238 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.869+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e100b2830 con 0x7f6e100b0860 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2885813745 0 0) 0x7f6df4004eb0 con 0x7f6e100af8e0 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e100b2b10 con 0x7f6e100af8e0 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f6e0c001850 con 0x7f6e100af8e0 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 10 monclient(hunting): dump: 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3838740711 0 0) 0x7f6e100b2830 con 0x7f6e100b0860 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6df4004eb0 con 0x7f6e100b0860 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f6e00001b80 con 0x7f6e100b0860 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 10 monclient(hunting): dump: 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 186521711 0 0) 0x7f6e100b1400 con 0x7f6e1013ef30 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e100b2830 con 0x7f6e1013ef30 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2468076763 0 0) 0x7f6df4004eb0 con 0x7f6e100b0860 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.870+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 >> v1:172.21.15.189:6789/0 conn(0x7f6e1013ef30 legacy=0x7f6e101d3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 >> v1:172.21.15.98:6789/0 conn(0x7f6e100af8e0 legacy=0x7f6e100b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e05ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e05ffb640 1 monclient: found mon.c 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e05ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e100b4060 con 0x7f6e100b0860 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e05ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e05ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e176d9640 5 monclient: authenticate success, global_id 4238 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e176d9640 10 monclient: _renew_subs 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e176d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e176d9640 1 -- 172.21.15.98:0/3453622292 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e100b4310 con 0x7f6e100b0860 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e176d9640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e176d9640 10 monclient: _renew_subs 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e176d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e176d9640 1 -- 172.21.15.98:0/3453622292 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e10138330 con 0x7f6e100b0860 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.871+0000 7f6e176d9640 1 librados: init done 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.872+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e00001ce0 con 0x7f6e100b0860 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.872+0000 7f6e05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.872+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f6e000038a0 con 0x7f6e100b0860 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.872+0000 7f6e06ffd640 4 set_mon_vals no callback set 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.872+0000 7f6e05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.873+0000 7f6e05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.873+0000 7f6e05ffb640 10 monclient: dump: 2024-03-29T10:44:51.870 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:51.871 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:51.871 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:51.871 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:51.871 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:51.871 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:51.871 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:51.871 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:51.871 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:51.871 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:51.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.873+0000 7f6e176d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:51.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.873+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f6e00003ac0 con 0x7f6e100b0860 2024-03-29T10:44:51.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.873+0000 7f6e176d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:51.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.873+0000 7f6e176d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:44:51.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.873+0000 7f6e176d9640 1 -- 172.21.15.98:0/3453622292 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df0005080 con 0x7f6e100b0860 2024-03-29T10:44:51.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.874+0000 7f6e15475640 10 monclient: get_auth_request con 0x7f6df406e2a0 auth_method 0 2024-03-29T10:44:51.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.874+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f6e000849e0 con 0x7f6e100b0860 2024-03-29T10:44:51.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.874+0000 7f6e05ffb640 10 client.4238.objecter ms_dispatch 0x7f6e10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:51.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.874+0000 7f6e05ffb640 3 client.4238.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:51.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.874+0000 7f6e05ffb640 3 client.4238.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:51.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.874+0000 7f6e05ffb640 20 client.4238.objecter dump_active .. 0 homeless 2024-03-29T10:44:51.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.874+0000 7f6e05ffb640 10 client.4238.objecter ms_handle_connect 0x7f6df406e2a0 2024-03-29T10:44:51.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.887+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e000502c0 con 0x7f6e100b0860 2024-03-29T10:44:51.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.887+0000 7f6e05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:51.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:51.887+0000 7f6e05ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:52.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.251+0000 7f6e176d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:52.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.251+0000 7f6e176d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:52.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.251+0000 7f6e176d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:44:52.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.251+0000 7f6e176d9640 1 -- 172.21.15.98:0/3453622292 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6df0005330 con 0x7f6e100b0860 2024-03-29T10:44:52.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.252+0000 7f6e05ffb640 1 -- 172.21.15.98:0/3453622292 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f6e000521f0 con 0x7f6e100b0860 2024-03-29T10:44:52.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.252+0000 7f6e05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:52.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.252+0000 7f6e05ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:52.237 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:44:52.237 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:52.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.255+0000 7f6e176d9640 10 librados: watch_flush enter 2024-03-29T10:44:52.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.255+0000 7f6e176d9640 10 librados: watch_flush exit 2024-03-29T10:44:52.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.255+0000 7f6e176d9640 20 client.4238.objecter shutdown clearing up homeless session... 2024-03-29T10:44:52.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.255+0000 7f6e176d9640 10 client.4238.objecter successfully canceled tick 2024-03-29T10:44:52.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.255+0000 7f6e176d9640 1 -- 172.21.15.98:0/3453622292 >> v1:172.21.15.98:6816/1481799426 conn(0x7f6df406e2a0 legacy=0x7f6df4070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:52.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.255+0000 7f6e176d9640 10 monclient: shutdown 2024-03-29T10:44:52.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.255+0000 7f6e176d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:52.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.255+0000 7f6e176d9640 1 -- 172.21.15.98:0/3453622292 >> v1:172.21.15.98:6790/0 conn(0x7f6e100b0860 legacy=0x7f6e101d3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:52.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.255+0000 7f6e176d9640 1 -- 172.21.15.98:0/3453622292 shutdown_connections 2024-03-29T10:44:52.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.255+0000 7f6e176d9640 1 -- 172.21.15.98:0/3453622292 >> 172.21.15.98:0/3453622292 conn(0x7f6e1006b660 msgr2=0x7f6e1013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:52.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.256+0000 7f6e176d9640 1 -- 172.21.15.98:0/3453622292 shutdown_connections 2024-03-29T10:44:52.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.256+0000 7f6e176d9640 1 -- 172.21.15.98:0/3453622292 wait complete. 2024-03-29T10:44:52.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:52.256+0000 7f6e176d9640 1 librados: shutdown 2024-03-29T10:44:52.254 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:59.061 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:59.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.220+0000 7f16f7d6b640 10 monclient: get_monmap_and_config 2024-03-29T10:44:59.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.220+0000 7f16f7d6b640 10 monclient: build_initial_monmap 2024-03-29T10:44:59.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.220+0000 7f16f7d6b640 10 monclient: monmap: 2024-03-29T10:44:59.204 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:44:59.205 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:59.205 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:44:59.222700+0000 2024-03-29T10:44:59.205 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:44:59.222700+0000 2024-03-29T10:44:59.205 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:59.205 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:59.205 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:59.205 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:44:59.205 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:44:59.205 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:59.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.221+0000 7f16f7d6b640 1 Processor -- start 2024-03-29T10:44:59.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.222+0000 7f16f7d6b640 1 -- start start 2024-03-29T10:44:59.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.222+0000 7f16f7d6b640 10 monclient: init 2024-03-29T10:44:59.207 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.222+0000 7f16f7d6b640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.222+0000 7f16f7d6b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.222+0000 7f16f7d6b640 10 monclient(hunting): picked mon.noname-b con 0x7f16f01457f0 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.222+0000 7f16f7d6b640 10 monclient(hunting): picked mon.noname-a con 0x7f16f013f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.222+0000 7f16f7d6b640 10 monclient(hunting): picked mon.noname-c con 0x7f16f00a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.222+0000 7f16f7d6b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f00aaef0 con 0x7f16f013f650 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.222+0000 7f16f7d6b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f0145bd0 con 0x7f16f00a9d70 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.222+0000 7f16f7d6b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f0145eb0 con 0x7f16f01457f0 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.222+0000 7f16f7d6b640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.222+0000 7f16f7d6b640 10 monclient(hunting): authenticate will time out at 919.420929s 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.225+0000 7f16f5b07640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f16f01457f0 0x7f16f013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:51533/0 (socket says 172.21.15.98:51533) 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.225+0000 7f16f5b07640 1 -- 172.21.15.98:0/554896226 learned_addr learned my addr 172.21.15.98:0/554896226 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:44:59.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 1 -- 172.21.15.98:0/554896226 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325921900 0 0) 0x7f16f0145eb0 con 0x7f16f01457f0 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 10 monclient(hunting): _negotiate 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 1 -- 172.21.15.98:0/554896226 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16dc003520 con 0x7f16f01457f0 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 1 -- 172.21.15.98:0/554896226 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661673561 0 0) 0x7f16f0145bd0 con 0x7f16f00a9d70 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 10 monclient(hunting): _negotiate 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 1 -- 172.21.15.98:0/554896226 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f0145eb0 con 0x7f16f00a9d70 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 1 -- 172.21.15.98:0/554896226 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 815768635 0 0) 0x7f16dc003520 con 0x7f16f01457f0 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 1 -- 172.21.15.98:0/554896226 >> v1:172.21.15.189:6789/0 conn(0x7f16f00a9d70 legacy=0x7f16f014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:59.212 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.226+0000 7f16f4b05640 1 -- 172.21.15.98:0/554896226 >> v1:172.21.15.98:6789/0 conn(0x7f16f013f650 legacy=0x7f16f00b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 10 monclient: _finish_hunting 0 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 1 monclient: found mon.noname-b 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 1 -- 172.21.15.98:0/554896226 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f0146340 con 0x7f16f01457f0 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 10 monclient: _finish_auth 0 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 10 monclient: _check_auth_tickets 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f7d6b640 5 monclient: authenticate success, global_id 4241 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f7d6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 1 -- 172.21.15.98:0/554896226 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f16e8001a10 con 0x7f16f01457f0 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 10 monclient: dump: 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f7d6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 1 -- 172.21.15.98:0/554896226 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16e8001c30 con 0x7f16f01457f0 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f4b05640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f7d6b640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.229+0000 7f16f7d6b640 4 set_mon_vals no callback set 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.230+0000 7f16f7d6b640 10 monclient: shutdown 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.230+0000 7f16f7d6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.230+0000 7f16f7d6b640 1 -- 172.21.15.98:0/554896226 >> v1:172.21.15.98:6790/0 conn(0x7f16f01457f0 legacy=0x7f16f013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:59.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.230+0000 7f16f7d6b640 1 -- 172.21.15.98:0/554896226 shutdown_connections 2024-03-29T10:44:59.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.230+0000 7f16f7d6b640 1 -- 172.21.15.98:0/554896226 >> 172.21.15.98:0/554896226 conn(0x7f16f0066a00 msgr2=0x7f16f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:59.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.230+0000 7f16f7d6b640 1 -- 172.21.15.98:0/554896226 shutdown_connections 2024-03-29T10:44:59.219 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.231+0000 7f16f7d6b640 1 -- 172.21.15.98:0/554896226 wait complete. 2024-03-29T10:44:59.219 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.232+0000 7f16f7d6b640 10 monclient: build_initial_monmap 2024-03-29T10:44:59.219 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.232+0000 7f16f7d6b640 10 monclient: monmap: 2024-03-29T10:44:59.219 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:44:59.219 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:59.219 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:44:59.219 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:44:59.219 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:59.220 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:59.220 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:44:59.220 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:44:59.220 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:44:59.220 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:59.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.232+0000 7f16f7d6b640 1 librados: starting msgr at 2024-03-29T10:44:59.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.232+0000 7f16f7d6b640 1 librados: starting objecter 2024-03-29T10:44:59.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.232+0000 7f16f7d6b640 1 Processor -- start 2024-03-29T10:44:59.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.232+0000 7f16f7d6b640 1 -- start start 2024-03-29T10:44:59.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.232+0000 7f16f7d6b640 1 librados: setting wanted keys 2024-03-29T10:44:59.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.232+0000 7f16f7d6b640 1 librados: calling monclient init 2024-03-29T10:44:59.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.232+0000 7f16f7d6b640 10 monclient: init 2024-03-29T10:44:59.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.233+0000 7f16f7d6b640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:59.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.233+0000 7f16f7d6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:59.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.233+0000 7f16f7d6b640 10 monclient(hunting): picked mon.noname-a con 0x7f16f00a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T10:44:59.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.233+0000 7f16f7d6b640 10 monclient(hunting): picked mon.noname-c con 0x7f16f013f650 addr v1:172.21.15.98:6790/0 2024-03-29T10:44:59.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.233+0000 7f16f7d6b640 10 monclient(hunting): picked mon.noname-b con 0x7f16f01457f0 addr v1:172.21.15.189:6789/0 2024-03-29T10:44:59.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.233+0000 7f16f7d6b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f01d9080 con 0x7f16f00a9d70 2024-03-29T10:44:59.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.233+0000 7f16f7d6b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f0134a80 con 0x7f16f01457f0 2024-03-29T10:44:59.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.233+0000 7f16f7d6b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f01df070 con 0x7f16f013f650 2024-03-29T10:44:59.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.233+0000 7f16f7d6b640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:59.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.233+0000 7f16f7d6b640 10 monclient(hunting): authenticate will time out at 919.431598s 2024-03-29T10:44:59.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.234+0000 7f16f6308640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f16f01457f0 0x7f16f01e6c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:50226/0 (socket says 172.21.15.98:50226) 2024-03-29T10:44:59.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.234+0000 7f16f6308640 1 -- 172.21.15.98:0/2357735274 learned_addr learned my addr 172.21.15.98:0/2357735274 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:44:59.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.234+0000 7f16e67fc640 10 client.?.objecter ms_handle_connect 0x7f16f01457f0 2024-03-29T10:44:59.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.234+0000 7f16e67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:59.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.235+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705002231 0 0) 0x7f16f0134a80 con 0x7f16f01457f0 2024-03-29T10:44:59.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.235+0000 7f16e67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:59.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.235+0000 7f16e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:59.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.235+0000 7f16e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:59.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.235+0000 7f16e67fc640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:44:59.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.235+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d4004eb0 con 0x7f16f01457f0 2024-03-29T10:44:59.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.235+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3413771638 0 0) 0x7f16d4004eb0 con 0x7f16f01457f0 2024-03-29T10:44:59.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.235+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16f0134a80 con 0x7f16f01457f0 2024-03-29T10:44:59.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.235+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f16ec001990 con 0x7f16f01457f0 2024-03-29T10:44:59.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.235+0000 7f16e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:59.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.235+0000 7f16e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:59.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.235+0000 7f16e67fc640 10 monclient(hunting): dump: 2024-03-29T10:44:59.233 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:59.233 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:59.233 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:59.233 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:59.233 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:59.233 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:59.233 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:59.233 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:59.234 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:59.234 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:59.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.235+0000 7f16e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:59.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1253650094 0 0) 0x7f16f0134a80 con 0x7f16f01457f0 2024-03-29T10:44:59.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 >> v1:172.21.15.98:6790/0 conn(0x7f16f013f650 legacy=0x7f16f01db410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:59.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 >> v1:172.21.15.98:6789/0 conn(0x7f16f00a9d70 legacy=0x7f16f0134370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:59.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16e67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:59.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16e67fc640 1 monclient: found mon.b 2024-03-29T10:44:59.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:59.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16e67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:59.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f01dbb20 con 0x7f16f01457f0 2024-03-29T10:44:59.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16e67fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16e67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16f7d6b640 5 monclient: authenticate success, global_id 4246 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16f7d6b640 10 monclient: _renew_subs 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16f7d6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16f7d6b640 1 -- 172.21.15.98:0/2357735274 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f01df590 con 0x7f16f01457f0 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16f7d6b640 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16f7d6b640 10 monclient: _renew_subs 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16f7d6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16f7d6b640 1 -- 172.21.15.98:0/2357735274 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f0145180 con 0x7f16f01457f0 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.237+0000 7f16f7d6b640 1 librados: init done 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.239+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16ec001c90 con 0x7f16f01457f0 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.239+0000 7f16e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:59.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.239+0000 7f16e7fff640 4 set_mon_vals no callback set 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.239+0000 7f16f7d6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.240+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f16ec003a60 con 0x7f16f01457f0 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.240+0000 7f16f7d6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.240+0000 7f16f7d6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.240+0000 7f16f7d6b640 1 -- 172.21.15.98:0/2357735274 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d8005080 con 0x7f16f01457f0 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.240+0000 7f16e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.241+0000 7f16e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.241+0000 7f16e67fc640 10 monclient: dump: 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:59.236 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.241+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f16ec003bc0 con 0x7f16f01457f0 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.242+0000 7f16f5b07640 10 monclient: get_auth_request con 0x7f16d406dac0 auth_method 0 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.243+0000 7f16e67fc640 10 client.4246.objecter ms_handle_connect 0x7f16d406dac0 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.243+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f16ec084b20 con 0x7f16f01457f0 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.243+0000 7f16e67fc640 10 client.4246.objecter ms_dispatch 0x7f16f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.243+0000 7f16e67fc640 3 client.4246.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.243+0000 7f16e67fc640 3 client.4246.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.243+0000 7f16e67fc640 20 client.4246.objecter dump_active .. 0 homeless 2024-03-29T10:44:59.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.250+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16ec050400 con 0x7f16f01457f0 2024-03-29T10:44:59.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.250+0000 7f16e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:59.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.250+0000 7f16e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:59.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.566+0000 7f16f7d6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:59.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.566+0000 7f16f7d6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:59.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.566+0000 7f16f7d6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:44:59.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.566+0000 7f16f7d6b640 1 -- 172.21.15.98:0/2357735274 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16d8005330 con 0x7f16f01457f0 2024-03-29T10:44:59.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.569+0000 7f16e67fc640 1 -- 172.21.15.98:0/2357735274 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f16ec052330 con 0x7f16f01457f0 2024-03-29T10:44:59.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.569+0000 7f16e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:59.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.569+0000 7f16e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:59.553 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:44:59.553 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:59.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.572+0000 7f16f7d6b640 10 librados: watch_flush enter 2024-03-29T10:44:59.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.572+0000 7f16f7d6b640 10 librados: watch_flush exit 2024-03-29T10:44:59.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.572+0000 7f16f7d6b640 20 client.4246.objecter shutdown clearing up homeless session... 2024-03-29T10:44:59.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.572+0000 7f16f7d6b640 10 client.4246.objecter successfully canceled tick 2024-03-29T10:44:59.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.573+0000 7f16f7d6b640 1 -- 172.21.15.98:0/2357735274 >> v1:172.21.15.98:6816/1481799426 conn(0x7f16d406dac0 legacy=0x7f16d406ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:59.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.573+0000 7f16f7d6b640 10 monclient: shutdown 2024-03-29T10:44:59.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.573+0000 7f16f7d6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:59.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.573+0000 7f16f7d6b640 1 -- 172.21.15.98:0/2357735274 >> v1:172.21.15.189:6789/0 conn(0x7f16f01457f0 legacy=0x7f16f01e6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:59.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.573+0000 7f16f7d6b640 1 -- 172.21.15.98:0/2357735274 shutdown_connections 2024-03-29T10:44:59.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.573+0000 7f16f7d6b640 1 -- 172.21.15.98:0/2357735274 >> 172.21.15.98:0/2357735274 conn(0x7f16f0066a00 msgr2=0x7f16f00b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:59.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.574+0000 7f16f7d6b640 1 -- 172.21.15.98:0/2357735274 shutdown_connections 2024-03-29T10:44:59.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.574+0000 7f16f7d6b640 1 -- 172.21.15.98:0/2357735274 wait complete. 2024-03-29T10:44:59.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:44:59.574+0000 7f16f7d6b640 1 librados: shutdown 2024-03-29T10:44:59.574 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:06.381 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:06.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.544+0000 7f92abe9a640 10 monclient: get_monmap_and_config 2024-03-29T10:45:06.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.544+0000 7f92abe9a640 10 monclient: build_initial_monmap 2024-03-29T10:45:06.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.545+0000 7f92abe9a640 10 monclient: monmap: 2024-03-29T10:45:06.528 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:06.528 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:06.528 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:45:06.545822+0000 2024-03-29T10:45:06.528 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:45:06.545822+0000 2024-03-29T10:45:06.528 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:06.528 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:06.528 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:06.528 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:45:06.528 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:45:06.528 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:06.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.545+0000 7f92abe9a640 1 Processor -- start 2024-03-29T10:45:06.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.546+0000 7f92abe9a640 1 -- start start 2024-03-29T10:45:06.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.546+0000 7f92abe9a640 10 monclient: init 2024-03-29T10:45:06.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.547+0000 7f92abe9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:06.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.547+0000 7f92abe9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:06.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.547+0000 7f92abe9a640 10 monclient(hunting): picked mon.noname-b con 0x7f92a41457f0 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:06.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.547+0000 7f92abe9a640 10 monclient(hunting): picked mon.noname-a con 0x7f92a413f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:06.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.547+0000 7f92abe9a640 10 monclient(hunting): picked mon.noname-c con 0x7f92a40a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:06.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.547+0000 7f92abe9a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a40aaef0 con 0x7f92a413f650 2024-03-29T10:45:06.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.547+0000 7f92abe9a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a4145bd0 con 0x7f92a40a9d70 2024-03-29T10:45:06.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.547+0000 7f92abe9a640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a4145eb0 con 0x7f92a41457f0 2024-03-29T10:45:06.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.547+0000 7f92abe9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:06.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.547+0000 7f92abe9a640 10 monclient(hunting): authenticate will time out at 926.744416s 2024-03-29T10:45:06.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.550+0000 7f92aa437640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f92a40a9d70 0x7f92a414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:33241/0 (socket says 172.21.15.98:33241) 2024-03-29T10:45:06.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.550+0000 7f92aa437640 1 -- 172.21.15.98:0/2229806017 learned_addr learned my addr 172.21.15.98:0/2229806017 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:06.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.551+0000 7f92a8c34640 1 -- 172.21.15.98:0/2229806017 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646438804 0 0) 0x7f92a4145bd0 con 0x7f92a40a9d70 2024-03-29T10:45:06.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.551+0000 7f92a8c34640 10 monclient(hunting): _negotiate 2024-03-29T10:45:06.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.551+0000 7f92a8c34640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:06.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.551+0000 7f92a8c34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:06.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.551+0000 7f92a8c34640 10 monclient(hunting): my global_id is 4249 2024-03-29T10:45:06.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.551+0000 7f92a8c34640 1 -- 172.21.15.98:0/2229806017 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9290003520 con 0x7f92a40a9d70 2024-03-29T10:45:06.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 1 -- 172.21.15.98:0/2229806017 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2866624479 0 0) 0x7f9290003520 con 0x7f92a40a9d70 2024-03-29T10:45:06.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 1 -- 172.21.15.98:0/2229806017 >> v1:172.21.15.98:6790/0 conn(0x7f92a41457f0 legacy=0x7f92a413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:06.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 1 -- 172.21.15.98:0/2229806017 >> v1:172.21.15.98:6789/0 conn(0x7f92a413f650 legacy=0x7f92a40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:06.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 10 monclient: _finish_hunting 0 2024-03-29T10:45:06.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 1 monclient: found mon.noname-c 2024-03-29T10:45:06.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:06.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:45:06.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 1 -- 172.21.15.98:0/2229806017 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a4146340 con 0x7f92a40a9d70 2024-03-29T10:45:06.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 10 monclient: _finish_auth 0 2024-03-29T10:45:06.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 10 monclient: _check_auth_tickets 2024-03-29T10:45:06.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:06.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92abe9a640 5 monclient: authenticate success, global_id 4249 2024-03-29T10:45:06.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92abe9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:06.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 1 -- 172.21.15.98:0/2229806017 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f92a0001690 con 0x7f92a40a9d70 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 10 monclient: dump: 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:06.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92abe9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 1 -- 172.21.15.98:0/2229806017 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92a0001860 con 0x7f92a40a9d70 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92a8c34640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92abe9a640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.553+0000 7f92abe9a640 4 set_mon_vals no callback set 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.554+0000 7f92a8c34640 1 -- 172.21.15.98:0/2229806017 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f92a0001cb0 con 0x7f92a40a9d70 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.554+0000 7f92a8c34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.554+0000 7f92a8c34640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.554+0000 7f92a8c34640 10 monclient: dump: 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:06.539 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:06.540 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:06.540 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:06.540 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:06.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.554+0000 7f92abe9a640 10 monclient: shutdown 2024-03-29T10:45:06.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.554+0000 7f92abe9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:06.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.554+0000 7f92abe9a640 1 -- 172.21.15.98:0/2229806017 >> v1:172.21.15.189:6789/0 conn(0x7f92a40a9d70 legacy=0x7f92a414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:06.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.557+0000 7f92abe9a640 1 -- 172.21.15.98:0/2229806017 shutdown_connections 2024-03-29T10:45:06.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.557+0000 7f92abe9a640 1 -- 172.21.15.98:0/2229806017 >> 172.21.15.98:0/2229806017 conn(0x7f92a4066a00 msgr2=0x7f92a4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:06.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.557+0000 7f92abe9a640 1 -- 172.21.15.98:0/2229806017 shutdown_connections 2024-03-29T10:45:06.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.558+0000 7f92abe9a640 1 -- 172.21.15.98:0/2229806017 wait complete. 2024-03-29T10:45:06.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.558+0000 7f92abe9a640 10 monclient: build_initial_monmap 2024-03-29T10:45:06.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.558+0000 7f92abe9a640 10 monclient: monmap: 2024-03-29T10:45:06.542 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:06.542 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:06.542 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:45:06.542 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:45:06.542 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:06.542 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:06.542 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:06.542 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:45:06.543 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:45:06.543 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:06.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.559+0000 7f92abe9a640 1 librados: starting msgr at 2024-03-29T10:45:06.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.559+0000 7f92abe9a640 1 librados: starting objecter 2024-03-29T10:45:06.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.559+0000 7f92abe9a640 1 Processor -- start 2024-03-29T10:45:06.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.559+0000 7f92abe9a640 1 -- start start 2024-03-29T10:45:06.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.559+0000 7f92abe9a640 1 librados: setting wanted keys 2024-03-29T10:45:06.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.559+0000 7f92abe9a640 1 librados: calling monclient init 2024-03-29T10:45:06.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.559+0000 7f92abe9a640 10 monclient: init 2024-03-29T10:45:06.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.560+0000 7f92abe9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:06.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.560+0000 7f92abe9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:06.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.560+0000 7f92abe9a640 10 monclient(hunting): picked mon.noname-c con 0x7f92a40a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:06.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.560+0000 7f92abe9a640 10 monclient(hunting): picked mon.noname-b con 0x7f92a413f650 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:06.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.560+0000 7f92abe9a640 10 monclient(hunting): picked mon.noname-a con 0x7f92a41457f0 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:06.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.560+0000 7f92abe9a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a406ed60 con 0x7f92a41457f0 2024-03-29T10:45:06.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.560+0000 7f92abe9a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a41dc040 con 0x7f92a413f650 2024-03-29T10:45:06.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.560+0000 7f92abe9a640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a4071f80 con 0x7f92a40a9d70 2024-03-29T10:45:06.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.560+0000 7f92abe9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:06.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.560+0000 7f92abe9a640 10 monclient(hunting): authenticate will time out at 926.757458s 2024-03-29T10:45:06.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.567+0000 7f92a9435640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f92a413f650 0x7f92a413fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:33339/0 (socket says 172.21.15.98:33339) 2024-03-29T10:45:06.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.567+0000 7f92a9435640 1 -- 172.21.15.98:0/755174476 learned_addr learned my addr 172.21.15.98:0/755174476 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:06.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.567+0000 7f92967fc640 10 client.?.objecter ms_handle_connect 0x7f92a413f650 2024-03-29T10:45:06.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.567+0000 7f92967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:06.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.567+0000 7f92967fc640 10 client.?.objecter ms_handle_connect 0x7f92a40a9d70 2024-03-29T10:45:06.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.567+0000 7f92967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:06.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.567+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149836166 0 0) 0x7f92a41dc040 con 0x7f92a413f650 2024-03-29T10:45:06.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.567+0000 7f92967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.567+0000 7f92967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.567+0000 7f92967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.567+0000 7f92967fc640 10 monclient(hunting): my global_id is 4252 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.567+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9288004eb0 con 0x7f92a413f650 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.567+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199373119 0 0) 0x7f92a4071f80 con 0x7f92a40a9d70 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 10 monclient(hunting): my global_id is 4244 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a41dc040 con 0x7f92a40a9d70 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1075704703 0 0) 0x7f92a41dc040 con 0x7f92a40a9d70 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92a4071f80 con 0x7f92a40a9d70 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f929c001b10 con 0x7f92a40a9d70 2024-03-29T10:45:06.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 10 monclient(hunting): dump: 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 798608537 0 0) 0x7f92a4071f80 con 0x7f92a40a9d70 2024-03-29T10:45:06.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 >> v1:172.21.15.189:6789/0 conn(0x7f92a413f650 legacy=0x7f92a413fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:06.554 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.568+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 >> v1:172.21.15.98:6789/0 conn(0x7f92a41457f0 legacy=0x7f92a41e6200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.569+0000 7f92967fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.569+0000 7f92967fc640 1 monclient: found mon.c 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.569+0000 7f92967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.569+0000 7f92967fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.569+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a4072110 con 0x7f92a40a9d70 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.570+0000 7f92967fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.570+0000 7f92967fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.570+0000 7f92967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.570+0000 7f92abe9a640 5 monclient: authenticate success, global_id 4244 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.570+0000 7f92abe9a640 10 monclient: _renew_subs 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.570+0000 7f92abe9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.570+0000 7f92abe9a640 1 -- 172.21.15.98:0/755174476 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a41dc4d0 con 0x7f92a40a9d70 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.570+0000 7f92abe9a640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.570+0000 7f92abe9a640 10 monclient: _renew_subs 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.570+0000 7f92abe9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.570+0000 7f92abe9a640 1 -- 172.21.15.98:0/755174476 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a4140690 con 0x7f92a40a9d70 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.570+0000 7f92abe9a640 1 librados: init done 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.571+0000 7f92abe9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.571+0000 7f92abe9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.571+0000 7f92abe9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.571+0000 7f92abe9a640 1 -- 172.21.15.98:0/755174476 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a4075f10 con 0x7f92a40a9d70 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.572+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f929c002020 con 0x7f92a40a9d70 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.572+0000 7f92967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.572+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f929c003b10 con 0x7f92a40a9d70 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.572+0000 7f9297fff640 4 set_mon_vals no callback set 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.572+0000 7f92967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.572+0000 7f92967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.572+0000 7f92967fc640 10 monclient: dump: 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:06.558 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:06.559 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:06.559 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:06.559 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:06.559 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:06.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.575+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f929c019ae0 con 0x7f92a40a9d70 2024-03-29T10:45:06.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.576+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f929c083a70 con 0x7f92a40a9d70 2024-03-29T10:45:06.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.576+0000 7f92967fc640 10 client.4244.objecter ms_dispatch 0x7f92a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:06.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.576+0000 7f92967fc640 3 client.4244.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:06.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.576+0000 7f92967fc640 3 client.4244.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:06.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.576+0000 7f92967fc640 20 client.4244.objecter dump_active .. 0 homeless 2024-03-29T10:45:06.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.576+0000 7f92a9435640 10 monclient: get_auth_request con 0x7f928806dd90 auth_method 0 2024-03-29T10:45:06.560 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.577+0000 7f92967fc640 10 client.4244.objecter ms_handle_connect 0x7f928806dd90 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.582+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f929c04f180 con 0x7f92a40a9d70 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.582+0000 7f92967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.582+0000 7f92967fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:06.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.910+0000 7f92abe9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:06.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.910+0000 7f92abe9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:06.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.910+0000 7f92abe9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:06.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.910+0000 7f92abe9a640 1 -- 172.21.15.98:0/755174476 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92a413c600 con 0x7f92a40a9d70 2024-03-29T10:45:06.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.912+0000 7f92967fc640 1 -- 172.21.15.98:0/755174476 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f929c051280 con 0x7f92a40a9d70 2024-03-29T10:45:06.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.912+0000 7f92967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:06.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.912+0000 7f92967fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:06.897 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:45:06.897 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:06.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.917+0000 7f92abe9a640 10 librados: watch_flush enter 2024-03-29T10:45:06.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.917+0000 7f92abe9a640 10 librados: watch_flush exit 2024-03-29T10:45:06.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.917+0000 7f92abe9a640 20 client.4244.objecter shutdown clearing up homeless session... 2024-03-29T10:45:06.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.917+0000 7f92abe9a640 10 client.4244.objecter successfully canceled tick 2024-03-29T10:45:06.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.917+0000 7f92abe9a640 1 -- 172.21.15.98:0/755174476 >> v1:172.21.15.98:6816/1481799426 conn(0x7f928806dd90 legacy=0x7f9288070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:06.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.917+0000 7f92abe9a640 10 monclient: shutdown 2024-03-29T10:45:06.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.917+0000 7f92abe9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:06.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.917+0000 7f92abe9a640 1 -- 172.21.15.98:0/755174476 >> v1:172.21.15.98:6790/0 conn(0x7f92a40a9d70 legacy=0x7f92a41db7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:06.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.919+0000 7f92abe9a640 1 -- 172.21.15.98:0/755174476 shutdown_connections 2024-03-29T10:45:06.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.919+0000 7f92abe9a640 1 -- 172.21.15.98:0/755174476 >> 172.21.15.98:0/755174476 conn(0x7f92a4066a00 msgr2=0x7f92a40b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:06.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.920+0000 7f92abe9a640 1 -- 172.21.15.98:0/755174476 shutdown_connections 2024-03-29T10:45:06.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.920+0000 7f92abe9a640 1 -- 172.21.15.98:0/755174476 wait complete. 2024-03-29T10:45:06.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:06.920+0000 7f92abe9a640 1 librados: shutdown 2024-03-29T10:45:06.921 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:13.728 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:13.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.896+0000 7f5acac3b640 10 monclient: get_monmap_and_config 2024-03-29T10:45:13.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.896+0000 7f5acac3b640 10 monclient: build_initial_monmap 2024-03-29T10:45:13.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.896+0000 7f5acac3b640 10 monclient: monmap: 2024-03-29T10:45:13.882 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:13.882 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:13.882 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:45:13.897805+0000 2024-03-29T10:45:13.882 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:45:13.897805+0000 2024-03-29T10:45:13.882 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:13.882 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:13.882 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:13.882 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:45:13.882 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:45:13.882 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:13.882 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.899+0000 7f5acac3b640 1 Processor -- start 2024-03-29T10:45:13.883 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.899+0000 7f5acac3b640 1 -- start start 2024-03-29T10:45:13.883 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.899+0000 7f5acac3b640 10 monclient: init 2024-03-29T10:45:13.883 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.900+0000 7f5acac3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:13.884 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.900+0000 7f5acac3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:13.884 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.900+0000 7f5acac3b640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac40a9ac0 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:13.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.900+0000 7f5acac3b640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac413ee90 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:13.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.903+0000 7f5acac3b640 10 monclient(hunting): picked mon.noname-b con 0x7f5ac40af900 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:13.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.904+0000 7f5ac3fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5ac413ee90 0x7f5ac413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:40077/0 (socket says 172.21.15.98:40077) 2024-03-29T10:45:13.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.907+0000 7f5ac3fff640 1 -- 172.21.15.98:0/259254000 learned_addr learned my addr 172.21.15.98:0/259254000 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:13.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.904+0000 7f5ac89d7640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5ac40a9ac0 0x7f5ac40adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33245/0 (socket says 172.21.15.98:33245) 2024-03-29T10:45:13.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.907+0000 7f5acac3b640 1 -- 172.21.15.98:0/259254000 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac40afd10 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.907+0000 7f5acac3b640 1 -- 172.21.15.98:0/259254000 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac40af6f0 con 0x7f5ac413ee90 2024-03-29T10:45:13.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.907+0000 7f5acac3b640 1 -- 172.21.15.98:0/259254000 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac40aac80 con 0x7f5ac40af900 2024-03-29T10:45:13.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.907+0000 7f5acac3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:13.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.907+0000 7f5acac3b640 10 monclient(hunting): authenticate will time out at 934.104619s 2024-03-29T10:45:13.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 1 -- 172.21.15.98:0/259254000 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767360060 0 0) 0x7f5ac40afd10 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:13.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 10 monclient(hunting): my global_id is 4251 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 1 -- 172.21.15.98:0/259254000 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aac005730 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 1 -- 172.21.15.98:0/259254000 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222789796 0 0) 0x7f5ac40af6f0 con 0x7f5ac413ee90 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 1 -- 172.21.15.98:0/259254000 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac40afd10 con 0x7f5ac413ee90 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 1 -- 172.21.15.98:0/259254000 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750117521 0 0) 0x7f5ac40aac80 con 0x7f5ac40af900 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.908+0000 7f5ac37fe640 1 -- 172.21.15.98:0/259254000 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac40af6f0 con 0x7f5ac40af900 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.910+0000 7f5ac37fe640 1 -- 172.21.15.98:0/259254000 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1784573754 0 0) 0x7f5aac005730 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.910+0000 7f5ac37fe640 1 -- 172.21.15.98:0/259254000 >> v1:172.21.15.98:6790/0 conn(0x7f5ac40af900 legacy=0x7f5ac4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.910+0000 7f5ac37fe640 1 -- 172.21.15.98:0/259254000 >> v1:172.21.15.189:6789/0 conn(0x7f5ac413ee90 legacy=0x7f5ac413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.910+0000 7f5ac37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.910+0000 7f5ac37fe640 1 monclient: found mon.noname-a 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.910+0000 7f5ac37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.910+0000 7f5ac37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.910+0000 7f5ac37fe640 1 -- 172.21.15.98:0/259254000 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac40aaf60 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.910+0000 7f5ac37fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.910+0000 7f5ac37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.910+0000 7f5ac37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.911+0000 7f5ac37fe640 1 -- 172.21.15.98:0/259254000 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5abc0019f0 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.911+0000 7f5ac37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.911+0000 7f5ac37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.911+0000 7f5ac37fe640 10 monclient: dump: 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.911+0000 7f5ac37fe640 1 -- 172.21.15.98:0/259254000 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5abc001bd0 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.911+0000 7f5ac37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.911+0000 7f5ac37fe640 1 -- 172.21.15.98:0/259254000 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5abc0019f0 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.911+0000 7f5ac37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.911+0000 7f5ac37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.911+0000 7f5ac37fe640 10 monclient: dump: 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:13.903 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:13.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.913+0000 7f5acac3b640 5 monclient: authenticate success, global_id 4251 2024-03-29T10:45:13.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.913+0000 7f5acac3b640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:13.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.913+0000 7f5acac3b640 4 set_mon_vals no callback set 2024-03-29T10:45:13.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.914+0000 7f5acac3b640 10 monclient: shutdown 2024-03-29T10:45:13.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.914+0000 7f5acac3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:13.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.914+0000 7f5acac3b640 1 -- 172.21.15.98:0/259254000 >> v1:172.21.15.98:6789/0 conn(0x7f5ac40a9ac0 legacy=0x7f5ac40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:13.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.914+0000 7f5acac3b640 1 -- 172.21.15.98:0/259254000 shutdown_connections 2024-03-29T10:45:13.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.914+0000 7f5acac3b640 1 -- 172.21.15.98:0/259254000 >> 172.21.15.98:0/259254000 conn(0x7f5ac40b35e0 msgr2=0x7f5ac40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:13.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.915+0000 7f5acac3b640 1 -- 172.21.15.98:0/259254000 shutdown_connections 2024-03-29T10:45:13.904 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.915+0000 7f5acac3b640 1 -- 172.21.15.98:0/259254000 wait complete. 2024-03-29T10:45:13.904 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.916+0000 7f5acac3b640 10 monclient: build_initial_monmap 2024-03-29T10:45:13.904 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.916+0000 7f5acac3b640 10 monclient: monmap: 2024-03-29T10:45:13.904 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:13.904 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:13.904 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:45:13.904 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:45:13.904 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:13.905 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:13.905 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:13.905 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:45:13.905 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:45:13.905 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:13.905 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.917+0000 7f5acac3b640 1 librados: starting msgr at 2024-03-29T10:45:13.905 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.919+0000 7f5acac3b640 1 librados: starting objecter 2024-03-29T10:45:13.905 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.919+0000 7f5acac3b640 1 Processor -- start 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.920+0000 7f5acac3b640 1 -- start start 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.920+0000 7f5acac3b640 1 librados: setting wanted keys 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.920+0000 7f5acac3b640 1 librados: calling monclient init 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.920+0000 7f5acac3b640 10 monclient: init 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.923+0000 7f5acac3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.923+0000 7f5acac3b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:13.907 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.923+0000 7f5acac3b640 10 monclient(hunting): picked mon.noname-b con 0x7f5ac40a9ac0 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:13.907 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.925+0000 7f5acac3b640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac40af900 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:13.910 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.926+0000 7f5acac3b640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac413ee90 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.926+0000 7f5ac89d7640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5ac40a9ac0 0x7f5ac4070870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:40199/0 (socket says 172.21.15.98:40199) 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.926+0000 7f5ac89d7640 1 -- 172.21.15.98:0/1383619254 learned_addr learned my addr 172.21.15.98:0/1383619254 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.926+0000 7f5acac3b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac413d590 con 0x7f5ac40af900 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.926+0000 7f5acac3b640 1 -- 172.21.15.98:0/1383619254 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac41d72a0 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.926+0000 7f5acac3b640 1 -- 172.21.15.98:0/1383619254 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac40720a0 con 0x7f5ac413ee90 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.926+0000 7f5ac17fa640 10 client.?.objecter ms_handle_connect 0x7f5ac40a9ac0 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.926+0000 7f5ac17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.926+0000 7f5acac3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.926+0000 7f5acac3b640 10 monclient(hunting): authenticate will time out at 934.123838s 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.926+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358723026 0 0) 0x7f5ac41d72a0 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.927+0000 7f5ac17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.927+0000 7f5ac17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.927+0000 7f5ac17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.927+0000 7f5ac17fa640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.927+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab0004eb0 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.927+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3536324578 0 0) 0x7f5ab0004eb0 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.927+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ac41d72a0 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.927+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5abc0019f0 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.927+0000 7f5ac17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.927+0000 7f5ac17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.927+0000 7f5ac17fa640 10 monclient(hunting): dump: 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.927+0000 7f5ac17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.928+0000 7f5ac17fa640 10 client.?.objecter ms_handle_connect 0x7f5ac413ee90 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.928+0000 7f5ac17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.928+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1447896524 0 0) 0x7f5ac41d72a0 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.928+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 >> v1:172.21.15.98:6790/0 conn(0x7f5ac413ee90 legacy=0x7f5ac41d6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.928+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 >> v1:172.21.15.98:6789/0 conn(0x7f5ac40af900 legacy=0x7f5ac41d6480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.931+0000 7f5ac17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.931+0000 7f5ac17fa640 1 monclient: found mon.b 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.931+0000 7f5ac17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.931+0000 7f5ac17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.931+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac4072380 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.931+0000 7f5ac17fa640 10 monclient: _finish_auth 0 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.931+0000 7f5ac17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.931+0000 7f5ac17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.931+0000 7f5acac3b640 5 monclient: authenticate success, global_id 4258 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.931+0000 7f5acac3b640 10 monclient: _renew_subs 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.931+0000 7f5acac3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.931+0000 7f5acac3b640 1 -- 172.21.15.98:0/1383619254 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac413d7e0 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.932+0000 7f5acac3b640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.932+0000 7f5acac3b640 10 monclient: _renew_subs 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.932+0000 7f5acac3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.932+0000 7f5acac3b640 1 -- 172.21.15.98:0/1383619254 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac4135ce0 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.932+0000 7f5acac3b640 1 librados: init done 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.932+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5abc0050e0 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.932+0000 7f5ac17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:13.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.932+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5abc005530 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.932+0000 7f5ac17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:13.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.932+0000 7f5ac17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:13.918 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.932+0000 7f5ac17fa640 10 monclient: dump: 2024-03-29T10:45:13.918 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:13.918 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:13.918 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:13.918 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:13.918 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:13.918 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:13.918 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:13.918 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:13.919 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:13.919 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:13.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.933+0000 7f5acac3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:13.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.933+0000 7f5acac3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:13.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.934+0000 7f5acac3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:13.919 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.934+0000 7f5acac3b640 1 -- 172.21.15.98:0/1383619254 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac413da20 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.935+0000 7f5ac2ffd640 4 set_mon_vals no callback set 2024-03-29T10:45:13.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.935+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f5abc005690 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.936+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f5abc085170 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.936+0000 7f5ac17fa640 10 client.4258.objecter ms_dispatch 0x7f5ac4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:13.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.936+0000 7f5ac17fa640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:13.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.936+0000 7f5ac17fa640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:13.920 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.937+0000 7f5ac17fa640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-29T10:45:13.921 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.939+0000 7f5ac3fff640 10 monclient: get_auth_request con 0x7f5ab006dd60 auth_method 0 2024-03-29T10:45:13.925 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.941+0000 7f5ac17fa640 10 client.4258.objecter ms_handle_connect 0x7f5ab006dd60 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.946+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5abc050a50 con 0x7f5ac40a9ac0 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.946+0000 7f5ac17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:13.946+0000 7f5ac17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:14.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.272+0000 7f5acac3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:14.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.272+0000 7f5acac3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:14.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.272+0000 7f5acac3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:14.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.272+0000 7f5acac3b640 1 -- 172.21.15.98:0/1383619254 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ac4075f10 con 0x7f5ac40a9ac0 2024-03-29T10:45:14.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.273+0000 7f5ac17fa640 1 -- 172.21.15.98:0/1383619254 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f5abc052980 con 0x7f5ac40a9ac0 2024-03-29T10:45:14.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.273+0000 7f5ac17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:14.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.273+0000 7f5ac17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:14.256 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:45:14.256 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:14.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.277+0000 7f5acac3b640 10 librados: watch_flush enter 2024-03-29T10:45:14.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.277+0000 7f5acac3b640 10 librados: watch_flush exit 2024-03-29T10:45:14.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.277+0000 7f5acac3b640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-29T10:45:14.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.277+0000 7f5acac3b640 10 client.4258.objecter successfully canceled tick 2024-03-29T10:45:14.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.279+0000 7f5acac3b640 1 -- 172.21.15.98:0/1383619254 >> v1:172.21.15.98:6816/1481799426 conn(0x7f5ab006dd60 legacy=0x7f5ab0070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:14.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.279+0000 7f5acac3b640 10 monclient: shutdown 2024-03-29T10:45:14.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.279+0000 7f5acac3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:14.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.279+0000 7f5acac3b640 1 -- 172.21.15.98:0/1383619254 >> v1:172.21.15.189:6789/0 conn(0x7f5ac40a9ac0 legacy=0x7f5ac4070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:14.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.279+0000 7f5acac3b640 1 -- 172.21.15.98:0/1383619254 shutdown_connections 2024-03-29T10:45:14.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.279+0000 7f5acac3b640 1 -- 172.21.15.98:0/1383619254 >> 172.21.15.98:0/1383619254 conn(0x7f5ac40b35e0 msgr2=0x7f5ac40ad060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:14.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.279+0000 7f5acac3b640 1 -- 172.21.15.98:0/1383619254 shutdown_connections 2024-03-29T10:45:14.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.279+0000 7f5acac3b640 1 -- 172.21.15.98:0/1383619254 wait complete. 2024-03-29T10:45:14.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:14.279+0000 7f5acac3b640 1 librados: shutdown 2024-03-29T10:45:14.283 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:21.090 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:21.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.258+0000 7f74f8954640 10 monclient: get_monmap_and_config 2024-03-29T10:45:21.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.258+0000 7f74f8954640 10 monclient: build_initial_monmap 2024-03-29T10:45:21.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.258+0000 7f74f8954640 10 monclient: monmap: 2024-03-29T10:45:21.243 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:21.243 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:21.243 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:45:21.259623+0000 2024-03-29T10:45:21.243 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:45:21.259623+0000 2024-03-29T10:45:21.243 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:21.243 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:21.243 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:21.243 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:45:21.243 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:45:21.243 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.261+0000 7f74f8954640 1 Processor -- start 2024-03-29T10:45:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.261+0000 7f74f8954640 1 -- start start 2024-03-29T10:45:21.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.261+0000 7f74f8954640 10 monclient: init 2024-03-29T10:45:21.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.262+0000 7f74f8954640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:21.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.262+0000 7f74f8954640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.262+0000 7f74f8954640 10 monclient(hunting): picked mon.noname-a con 0x7f74f40ae250 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.262+0000 7f74f8954640 10 monclient(hunting): picked mon.noname-c con 0x7f74f413f970 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.262+0000 7f74f8954640 10 monclient(hunting): picked mon.noname-b con 0x7f74f4142800 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.262+0000 7f74f8954640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f40afe00 con 0x7f74f40ae250 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.262+0000 7f74f8954640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f4142c10 con 0x7f74f413f970 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.262+0000 7f74f8954640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f40afb20 con 0x7f74f4142800 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.262+0000 7f74f8954640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.262+0000 7f74f8954640 10 monclient(hunting): authenticate will time out at 941.459228s 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.264+0000 7f74f2d9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f74f4142800 0x7f74f414da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:49091/0 (socket says 172.21.15.98:49091) 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.264+0000 7f74f2d9d640 1 -- 172.21.15.98:0/3708373999 learned_addr learned my addr 172.21.15.98:0/3708373999 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.265+0000 7f74f159a640 1 -- 172.21.15.98:0/3708373999 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442338899 0 0) 0x7f74f40afb20 con 0x7f74f4142800 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.265+0000 7f74f159a640 10 monclient(hunting): _negotiate 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.265+0000 7f74f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:21.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.265+0000 7f74f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.265+0000 7f74f159a640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:45:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.265+0000 7f74f159a640 1 -- 172.21.15.98:0/3708373999 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74dc003520 con 0x7f74f4142800 2024-03-29T10:45:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.265+0000 7f74f159a640 1 -- 172.21.15.98:0/3708373999 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4242276235 0 0) 0x7f74dc003520 con 0x7f74f4142800 2024-03-29T10:45:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.265+0000 7f74f159a640 1 -- 172.21.15.98:0/3708373999 >> v1:172.21.15.189:6789/0 conn(0x7f74f413f970 legacy=0x7f74f41420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:21.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.265+0000 7f74f159a640 1 -- 172.21.15.98:0/3708373999 >> v1:172.21.15.98:6789/0 conn(0x7f74f40ae250 legacy=0x7f74f413f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f159a640 10 monclient: _finish_hunting 0 2024-03-29T10:45:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f159a640 1 monclient: found mon.noname-b 2024-03-29T10:45:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:45:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f159a640 1 -- 172.21.15.98:0/3708373999 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f40aab80 con 0x7f74f4142800 2024-03-29T10:45:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f159a640 10 monclient: _finish_auth 0 2024-03-29T10:45:21.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f159a640 10 monclient: _check_auth_tickets 2024-03-29T10:45:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f159a640 1 -- 172.21.15.98:0/3708373999 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f74e8001690 con 0x7f74f4142800 2024-03-29T10:45:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:21.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f159a640 10 monclient: dump: 2024-03-29T10:45:21.253 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:21.253 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:21.253 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:21.253 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:21.253 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:21.253 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:21.253 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:21.253 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:21.254 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:21.254 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:21.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f8954640 5 monclient: authenticate success, global_id 4253 2024-03-29T10:45:21.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.268+0000 7f74f8954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:21.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.271+0000 7f74f159a640 1 -- 172.21.15.98:0/3708373999 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74e8001860 con 0x7f74f4142800 2024-03-29T10:45:21.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.271+0000 7f74f159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:21.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.271+0000 7f74f159a640 1 -- 172.21.15.98:0/3708373999 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f74e8001cb0 con 0x7f74f4142800 2024-03-29T10:45:21.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.271+0000 7f74f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:21.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.271+0000 7f74f159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:21.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.271+0000 7f74f159a640 10 monclient: dump: 2024-03-29T10:45:21.256 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:21.256 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:21.256 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:21.256 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:21.256 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:21.256 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:21.256 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:21.256 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:21.257 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:21.257 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.272+0000 7f74f8954640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.272+0000 7f74f8954640 4 set_mon_vals no callback set 2024-03-29T10:45:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.273+0000 7f74f8954640 10 monclient: shutdown 2024-03-29T10:45:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.273+0000 7f74f8954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:21.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.273+0000 7f74f8954640 1 -- 172.21.15.98:0/3708373999 >> v1:172.21.15.98:6790/0 conn(0x7f74f4142800 legacy=0x7f74f414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.273+0000 7f74f8954640 1 -- 172.21.15.98:0/3708373999 shutdown_connections 2024-03-29T10:45:21.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.273+0000 7f74f8954640 1 -- 172.21.15.98:0/3708373999 >> 172.21.15.98:0/3708373999 conn(0x7f74f40b35e0 msgr2=0x7f74f40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:21.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.273+0000 7f74f8954640 1 -- 172.21.15.98:0/3708373999 shutdown_connections 2024-03-29T10:45:21.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.274+0000 7f74f8954640 1 -- 172.21.15.98:0/3708373999 wait complete. 2024-03-29T10:45:21.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.275+0000 7f74f8954640 10 monclient: build_initial_monmap 2024-03-29T10:45:21.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.275+0000 7f74f8954640 10 monclient: monmap: 2024-03-29T10:45:21.260 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:21.260 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:21.260 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:45:21.260 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:45:21.260 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:21.260 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:21.261 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:21.261 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:45:21.261 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:45:21.261 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:21.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.275+0000 7f74f8954640 1 librados: starting msgr at 2024-03-29T10:45:21.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.275+0000 7f74f8954640 1 librados: starting objecter 2024-03-29T10:45:21.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.275+0000 7f74f8954640 1 Processor -- start 2024-03-29T10:45:21.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.276+0000 7f74f8954640 1 -- start start 2024-03-29T10:45:21.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.276+0000 7f74f8954640 1 librados: setting wanted keys 2024-03-29T10:45:21.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.276+0000 7f74f8954640 1 librados: calling monclient init 2024-03-29T10:45:21.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.276+0000 7f74f8954640 10 monclient: init 2024-03-29T10:45:21.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.276+0000 7f74f8954640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:21.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.276+0000 7f74f8954640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:21.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.276+0000 7f74f8954640 10 monclient(hunting): picked mon.noname-a con 0x7f74f40ae250 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:21.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.277+0000 7f74f8954640 10 monclient(hunting): picked mon.noname-b con 0x7f74f413f970 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:21.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.277+0000 7f74f8954640 10 monclient(hunting): picked mon.noname-c con 0x7f74f4142800 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:21.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.280+0000 7f74f8954640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f4145d20 con 0x7f74f40ae250 2024-03-29T10:45:21.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.280+0000 7f74f8954640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f41da4e0 con 0x7f74f413f970 2024-03-29T10:45:21.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.280+0000 7f74f8954640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f41464b0 con 0x7f74f4142800 2024-03-29T10:45:21.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.280+0000 7f74f8954640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:21.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.280+0000 7f74f8954640 10 monclient(hunting): authenticate will time out at 941.477277s 2024-03-29T10:45:21.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.281+0000 7f74f259c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f74f40ae250 0x7f74f41daa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54851/0 (socket says 172.21.15.98:54851) 2024-03-29T10:45:21.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.281+0000 7f74f259c640 1 -- 172.21.15.98:0/96296543 learned_addr learned my addr 172.21.15.98:0/96296543 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 10 client.?.objecter ms_handle_connect 0x7f74f40ae250 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 10 client.?.objecter ms_handle_connect 0x7f74f4142800 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883030130 0 0) 0x7f74f4145d20 con 0x7f74f40ae250 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d4004eb0 con 0x7f74f40ae250 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198623119 0 0) 0x7f74f41464b0 con 0x7f74f4142800 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:21.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.284+0000 7f74e2ffd640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:45:21.270 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.280+0000 7f74f1d9b640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f74f413f970 0x7f74f4137e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:57635/0 (socket says 172.21.15.98:57635) 2024-03-29T10:45:21.270 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.285+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f4145d20 con 0x7f74f4142800 2024-03-29T10:45:21.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.285+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2625755828 0 0) 0x7f74f4145d20 con 0x7f74f4142800 2024-03-29T10:45:21.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.285+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74f41464b0 con 0x7f74f4142800 2024-03-29T10:45:21.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.285+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f74e8001b70 con 0x7f74f4142800 2024-03-29T10:45:21.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.285+0000 7f74e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:21.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.285+0000 7f74e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:21.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.285+0000 7f74e2ffd640 10 monclient(hunting): dump: 2024-03-29T10:45:21.271 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.285+0000 7f74e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 831984477 0 0) 0x7f74f41464b0 con 0x7f74f4142800 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 >> v1:172.21.15.189:6789/0 conn(0x7f74f413f970 legacy=0x7f74f4137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 >> v1:172.21.15.98:6789/0 conn(0x7f74f40ae250 legacy=0x7f74f41daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:21.272 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 1 monclient: found mon.c 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f413a4f0 con 0x7f74f4142800 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74e8001cd0 con 0x7f74f4142800 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f74e8004c30 con 0x7f74f4142800 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74e2ffd640 10 monclient: dump: 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:21.273 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74f0d99640 4 set_mon_vals no callback set 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74f8954640 5 monclient: authenticate success, global_id 4256 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74f8954640 10 monclient: _renew_subs 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74f8954640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74f8954640 1 -- 172.21.15.98:0/96296543 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f41db110 con 0x7f74f4142800 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74f8954640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:21.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74f8954640 10 monclient: _renew_subs 2024-03-29T10:45:21.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74f8954640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:21.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74f8954640 1 -- 172.21.15.98:0/96296543 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f4138730 con 0x7f74f4142800 2024-03-29T10:45:21.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.286+0000 7f74f8954640 1 librados: init done 2024-03-29T10:45:21.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.288+0000 7f74f8954640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:21.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.288+0000 7f74f8954640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:21.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.288+0000 7f74f8954640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:21.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.288+0000 7f74f8954640 1 -- 172.21.15.98:0/96296543 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f413a6e0 con 0x7f74f4142800 2024-03-29T10:45:21.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.289+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f74e801ac00 con 0x7f74f4142800 2024-03-29T10:45:21.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.290+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f74e80849d0 con 0x7f74f4142800 2024-03-29T10:45:21.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.290+0000 7f74e2ffd640 10 client.4256.objecter ms_dispatch 0x7f74f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:21.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.290+0000 7f74e2ffd640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:21.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.290+0000 7f74e2ffd640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:21.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.290+0000 7f74e2ffd640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-29T10:45:21.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.290+0000 7f74f259c640 10 monclient: get_auth_request con 0x7f74d406dcc0 auth_method 0 2024-03-29T10:45:21.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.290+0000 7f74e2ffd640 10 client.4256.objecter ms_handle_connect 0x7f74d406dcc0 2024-03-29T10:45:21.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.299+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74e80502b0 con 0x7f74f4142800 2024-03-29T10:45:21.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.299+0000 7f74e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:21.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.299+0000 7f74e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:21.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.625+0000 7f74f8954640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:21.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.625+0000 7f74f8954640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:21.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.625+0000 7f74f8954640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:21.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.625+0000 7f74f8954640 1 -- 172.21.15.98:0/96296543 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74f4075f10 con 0x7f74f4142800 2024-03-29T10:45:21.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.627+0000 7f74e2ffd640 1 -- 172.21.15.98:0/96296543 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f74e80521e0 con 0x7f74f4142800 2024-03-29T10:45:21.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.627+0000 7f74e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:21.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.627+0000 7f74e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:21.612 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:45:21.613 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:21.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.630+0000 7f74f8954640 10 librados: watch_flush enter 2024-03-29T10:45:21.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.630+0000 7f74f8954640 10 librados: watch_flush exit 2024-03-29T10:45:21.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.630+0000 7f74f8954640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-29T10:45:21.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.630+0000 7f74f8954640 10 client.4256.objecter successfully canceled tick 2024-03-29T10:45:21.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.630+0000 7f74f8954640 1 -- 172.21.15.98:0/96296543 >> v1:172.21.15.98:6816/1481799426 conn(0x7f74d406dcc0 legacy=0x7f74d4070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:21.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.630+0000 7f74f8954640 10 monclient: shutdown 2024-03-29T10:45:21.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.630+0000 7f74f8954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:21.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.630+0000 7f74f8954640 1 -- 172.21.15.98:0/96296543 >> v1:172.21.15.98:6790/0 conn(0x7f74f4142800 legacy=0x7f74f41442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:21.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.631+0000 7f74f8954640 1 -- 172.21.15.98:0/96296543 shutdown_connections 2024-03-29T10:45:21.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.631+0000 7f74f8954640 1 -- 172.21.15.98:0/96296543 >> 172.21.15.98:0/96296543 conn(0x7f74f40b35e0 msgr2=0x7f74f413e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:21.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.632+0000 7f74f8954640 1 -- 172.21.15.98:0/96296543 shutdown_connections 2024-03-29T10:45:21.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.632+0000 7f74f8954640 1 -- 172.21.15.98:0/96296543 wait complete. 2024-03-29T10:45:21.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:21.632+0000 7f74f8954640 1 librados: shutdown 2024-03-29T10:45:21.632 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:28.440 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:28.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.606+0000 7f244d99c640 10 monclient: get_monmap_and_config 2024-03-29T10:45:28.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.606+0000 7f244d99c640 10 monclient: build_initial_monmap 2024-03-29T10:45:28.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.606+0000 7f244d99c640 10 monclient: monmap: 2024-03-29T10:45:28.589 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:28.589 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:28.589 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:45:28.607140+0000 2024-03-29T10:45:28.589 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:45:28.607140+0000 2024-03-29T10:45:28.589 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:28.589 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:28.590 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:28.590 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:45:28.590 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:45:28.590 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:28.591 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.609+0000 7f244d99c640 1 Processor -- start 2024-03-29T10:45:28.591 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.609+0000 7f244d99c640 1 -- start start 2024-03-29T10:45:28.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.609+0000 7f244d99c640 10 monclient: init 2024-03-29T10:45:28.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.610+0000 7f244d99c640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:28.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.610+0000 7f244d99c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:28.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.610+0000 7f244d99c640 10 monclient(hunting): picked mon.noname-a con 0x7f2440033000 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:28.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.610+0000 7f244d99c640 10 monclient(hunting): picked mon.noname-c con 0x7f244002c230 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:28.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.610+0000 7f244d99c640 10 monclient(hunting): picked mon.noname-b con 0x7f2440006b00 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:28.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.610+0000 7f244d99c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244002b7e0 con 0x7f2440033000 2024-03-29T10:45:28.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.610+0000 7f244d99c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244002d210 con 0x7f244002c230 2024-03-29T10:45:28.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.610+0000 7f244d99c640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244002d4f0 con 0x7f2440006b00 2024-03-29T10:45:28.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.610+0000 7f244d99c640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:28.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.610+0000 7f244d99c640 10 monclient(hunting): authenticate will time out at 948.807461s 2024-03-29T10:45:28.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.616+0000 7f244d19b640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f2440006b00 0x7f24400d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:40939/0 (socket says 172.21.15.98:40939) 2024-03-29T10:45:28.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.616+0000 7f244d19b640 1 -- 172.21.15.98:0/1726511177 learned_addr learned my addr 172.21.15.98:0/1726511177 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:28.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.616+0000 7f2447fff640 1 --1- 172.21.15.98:0/1726511177 >> v1:172.21.15.189:6789/0 conn(0x7f244002c230 0x7f24400c79f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:36279/0 (socket says 172.21.15.98:36279) 2024-03-29T10:45:28.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.616+0000 7f24477fe640 1 -- 172.21.15.98:0/1726511177 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714663575 0 0) 0x7f244002d4f0 con 0x7f2440006b00 2024-03-29T10:45:28.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.616+0000 7f24477fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:28.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.616+0000 7f24477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:28.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.616+0000 7f24477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:28.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.616+0000 7f24477fe640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:45:28.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.616+0000 7f24477fe640 1 -- 172.21.15.98:0/1726511177 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2438003520 con 0x7f2440006b00 2024-03-29T10:45:28.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 1 -- 172.21.15.98:0/1726511177 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060386927 0 0) 0x7f244002d210 con 0x7f244002c230 2024-03-29T10:45:28.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:28.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:28.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:28.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 10 monclient(hunting): my global_id is 4261 2024-03-29T10:45:28.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 1 -- 172.21.15.98:0/1726511177 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f244002d4f0 con 0x7f244002c230 2024-03-29T10:45:28.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 1 -- 172.21.15.98:0/1726511177 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 64446213 0 0) 0x7f2438003520 con 0x7f2440006b00 2024-03-29T10:45:28.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 1 -- 172.21.15.98:0/1726511177 >> v1:172.21.15.189:6789/0 conn(0x7f244002c230 legacy=0x7f24400c79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:28.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 1 -- 172.21.15.98:0/1726511177 >> v1:172.21.15.98:6789/0 conn(0x7f2440033000 legacy=0x7f24400c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:28.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:28.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 1 monclient: found mon.noname-b 2024-03-29T10:45:28.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:28.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:45:28.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 1 -- 172.21.15.98:0/1726511177 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24400cad70 con 0x7f2440006b00 2024-03-29T10:45:28.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:28.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:28.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:28.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 1 -- 172.21.15.98:0/1726511177 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f243c001690 con 0x7f2440006b00 2024-03-29T10:45:28.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f24477fe640 10 monclient: dump: 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f244d99c640 5 monclient: authenticate success, global_id 4259 2024-03-29T10:45:28.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.617+0000 7f244d99c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:28.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.621+0000 7f24477fe640 1 -- 172.21.15.98:0/1726511177 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f243c0017f0 con 0x7f2440006b00 2024-03-29T10:45:28.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.621+0000 7f24477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:28.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.621+0000 7f24477fe640 1 -- 172.21.15.98:0/1726511177 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f243c001c00 con 0x7f2440006b00 2024-03-29T10:45:28.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.621+0000 7f24477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:28.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.621+0000 7f24477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:28.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.621+0000 7f24477fe640 10 monclient: dump: 2024-03-29T10:45:28.604 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:28.605 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:28.605 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:28.605 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:28.605 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:28.605 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:28.605 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:28.605 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:28.605 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:28.605 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:28.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.621+0000 7f244d99c640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:28.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.621+0000 7f244d99c640 4 set_mon_vals no callback set 2024-03-29T10:45:28.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.622+0000 7f244d99c640 10 monclient: shutdown 2024-03-29T10:45:28.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.622+0000 7f244d99c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:28.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.622+0000 7f244d99c640 1 -- 172.21.15.98:0/1726511177 >> v1:172.21.15.98:6790/0 conn(0x7f2440006b00 legacy=0x7f24400d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:28.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.625+0000 7f244d99c640 1 -- 172.21.15.98:0/1726511177 shutdown_connections 2024-03-29T10:45:28.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.625+0000 7f244d99c640 1 -- 172.21.15.98:0/1726511177 >> 172.21.15.98:0/1726511177 conn(0x7f24400058f0 msgr2=0x7f2440005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:28.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.626+0000 7f244d99c640 1 -- 172.21.15.98:0/1726511177 shutdown_connections 2024-03-29T10:45:28.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.629+0000 7f244d99c640 1 -- 172.21.15.98:0/1726511177 wait complete. 2024-03-29T10:45:28.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.630+0000 7f244d99c640 10 monclient: build_initial_monmap 2024-03-29T10:45:28.615 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.630+0000 7f244d99c640 10 monclient: monmap: 2024-03-29T10:45:28.615 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:28.615 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:28.615 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:45:28.615 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:45:28.615 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:28.615 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:28.615 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:28.615 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:45:28.616 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:45:28.616 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:28.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.630+0000 7f244d99c640 1 librados: starting msgr at 2024-03-29T10:45:28.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.630+0000 7f244d99c640 1 librados: starting objecter 2024-03-29T10:45:28.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.630+0000 7f244d99c640 1 Processor -- start 2024-03-29T10:45:28.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.631+0000 7f244d99c640 1 -- start start 2024-03-29T10:45:28.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.631+0000 7f244d99c640 1 librados: setting wanted keys 2024-03-29T10:45:28.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.631+0000 7f244d99c640 1 librados: calling monclient init 2024-03-29T10:45:28.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.631+0000 7f244d99c640 10 monclient: init 2024-03-29T10:45:28.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.632+0000 7f244d99c640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:28.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.632+0000 7f244d99c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:28.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.632+0000 7f244d99c640 10 monclient(hunting): picked mon.noname-a con 0x7f2440006b00 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:28.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.635+0000 7f244d99c640 10 monclient(hunting): picked mon.noname-b con 0x7f244002c230 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:28.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.638+0000 7f244c99a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2440006b00 0x7f24400b9e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55497/0 (socket says 172.21.15.98:55497) 2024-03-29T10:45:28.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.639+0000 7f2447fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f244002c230 0x7f24400c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:36411/0 (socket says 172.21.15.98:36411) 2024-03-29T10:45:28.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.639+0000 7f244d99c640 10 monclient(hunting): picked mon.noname-c con 0x7f24400ba580 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:28.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.639+0000 7f244d99c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24400c40a0 con 0x7f2440006b00 2024-03-29T10:45:28.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.639+0000 7f244d99c640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24400c3050 con 0x7f244002c230 2024-03-29T10:45:28.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.639+0000 7f244d99c640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24400caa90 con 0x7f24400ba580 2024-03-29T10:45:28.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.639+0000 7f244d99c640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:28.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.639+0000 7f244d99c640 10 monclient(hunting): authenticate will time out at 948.836504s 2024-03-29T10:45:28.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.639+0000 7f244c99a640 1 -- 172.21.15.98:0/4155207539 learned_addr learned my addr 172.21.15.98:0/4155207539 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:28.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.639+0000 7f24457fa640 10 client.?.objecter ms_handle_connect 0x7f244002c230 2024-03-29T10:45:28.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.639+0000 7f24457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 10 client.?.objecter ms_handle_connect 0x7f24400ba580 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16585886 0 0) 0x7f24400caa90 con 0x7f24400ba580 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f242c004eb0 con 0x7f24400ba580 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 439387116 0 0) 0x7f242c004eb0 con 0x7f24400ba580 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24400caa90 con 0x7f24400ba580 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f243c001b00 con 0x7f24400ba580 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 10 monclient(hunting): dump: 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:28.625 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:28.626 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:28.626 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:28.626 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:28.626 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:28.626 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:28.626 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:28.626 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:28.626 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:28.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.640+0000 7f24457fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:28.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 134343409 0 0) 0x7f24400caa90 con 0x7f24400ba580 2024-03-29T10:45:28.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 >> v1:172.21.15.189:6789/0 conn(0x7f244002c230 legacy=0x7f24400c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:28.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 >> v1:172.21.15.98:6789/0 conn(0x7f2440006b00 legacy=0x7f24400b9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:28.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 10 monclient: _finish_hunting 0 2024-03-29T10:45:28.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 1 monclient: found mon.c 2024-03-29T10:45:28.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:28.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:28.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24400c2270 con 0x7f24400ba580 2024-03-29T10:45:28.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 10 monclient: _finish_auth 0 2024-03-29T10:45:28.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 10 monclient: _check_auth_tickets 2024-03-29T10:45:28.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:28.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f244d99c640 5 monclient: authenticate success, global_id 4262 2024-03-29T10:45:28.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f244d99c640 10 monclient: _renew_subs 2024-03-29T10:45:28.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f244d99c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:28.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f244d99c640 1 -- 172.21.15.98:0/4155207539 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24400ca440 con 0x7f24400ba580 2024-03-29T10:45:28.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f244d99c640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:28.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f244d99c640 10 monclient: _renew_subs 2024-03-29T10:45:28.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f244d99c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:28.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f244d99c640 1 -- 172.21.15.98:0/4155207539 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24400c7050 con 0x7f24400ba580 2024-03-29T10:45:28.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f244d99c640 1 librados: init done 2024-03-29T10:45:28.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f243c004a90 con 0x7f24400ba580 2024-03-29T10:45:28.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:28.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f243c004da0 con 0x7f24400ba580 2024-03-29T10:45:28.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:28.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:28.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.641+0000 7f24457fa640 10 monclient: dump: 2024-03-29T10:45:28.630 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:28.630 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:28.630 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:28.631 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:28.631 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:28.631 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:28.631 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:28.631 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:28.631 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:28.631 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:28.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.642+0000 7f2446ffd640 4 set_mon_vals no callback set 2024-03-29T10:45:28.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.642+0000 7f2432ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:28.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.643+0000 7f2432ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:28.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.643+0000 7f2432ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:28.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.643+0000 7f2432ffd640 1 -- 172.21.15.98:0/4155207539 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2420005080 con 0x7f24400ba580 2024-03-29T10:45:28.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.643+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f243c01ad70 con 0x7f24400ba580 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.644+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f243c084aa0 con 0x7f24400ba580 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.644+0000 7f24457fa640 10 client.4262.objecter ms_dispatch 0x7f2448000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.644+0000 7f24457fa640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.644+0000 7f24457fa640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.644+0000 7f244c99a640 10 monclient: get_auth_request con 0x7f242c06dcc0 auth_method 0 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.644+0000 7f24457fa640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.644+0000 7f24457fa640 10 client.4262.objecter ms_handle_connect 0x7f242c06dcc0 2024-03-29T10:45:28.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.652+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f243c0504c0 con 0x7f24400ba580 2024-03-29T10:45:28.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.652+0000 7f24457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:28.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.652+0000 7f24457fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:28.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.973+0000 7f2432ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:28.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.973+0000 7f2432ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:28.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.973+0000 7f2432ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:28.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.973+0000 7f2432ffd640 1 -- 172.21.15.98:0/4155207539 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2420005330 con 0x7f24400ba580 2024-03-29T10:45:28.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.974+0000 7f24457fa640 1 -- 172.21.15.98:0/4155207539 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f243c0523f0 con 0x7f24400ba580 2024-03-29T10:45:28.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.974+0000 7f24457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:28.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.974+0000 7f24457fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:28.959 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:45:28.959 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:28.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.980+0000 7f2432ffd640 10 librados: watch_flush enter 2024-03-29T10:45:28.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.980+0000 7f2432ffd640 10 librados: watch_flush exit 2024-03-29T10:45:28.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.980+0000 7f2432ffd640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T10:45:28.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.980+0000 7f2432ffd640 10 client.4262.objecter successfully canceled tick 2024-03-29T10:45:28.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.980+0000 7f2432ffd640 1 -- 172.21.15.98:0/4155207539 >> v1:172.21.15.98:6816/1481799426 conn(0x7f242c06dcc0 legacy=0x7f242c070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:28.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.980+0000 7f2432ffd640 10 monclient: shutdown 2024-03-29T10:45:28.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.980+0000 7f2432ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:28.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.980+0000 7f2432ffd640 1 -- 172.21.15.98:0/4155207539 >> v1:172.21.15.98:6790/0 conn(0x7f24400ba580 legacy=0x7f2440186370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:28.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.980+0000 7f2432ffd640 1 -- 172.21.15.98:0/4155207539 shutdown_connections 2024-03-29T10:45:28.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.980+0000 7f2432ffd640 1 -- 172.21.15.98:0/4155207539 >> 172.21.15.98:0/4155207539 conn(0x7f24400058f0 msgr2=0x7f2440033ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:28.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.984+0000 7f2432ffd640 1 -- 172.21.15.98:0/4155207539 shutdown_connections 2024-03-29T10:45:28.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.984+0000 7f2432ffd640 1 -- 172.21.15.98:0/4155207539 wait complete. 2024-03-29T10:45:28.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:28.984+0000 7f2432ffd640 1 librados: shutdown 2024-03-29T10:45:28.983 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:35.795 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:35.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.976+0000 7f544fd38640 10 monclient: get_monmap_and_config 2024-03-29T10:45:35.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.976+0000 7f544fd38640 10 monclient: build_initial_monmap 2024-03-29T10:45:35.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.976+0000 7f544fd38640 10 monclient: monmap: 2024-03-29T10:45:35.960 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:35.960 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:35.960 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:45:35.977763+0000 2024-03-29T10:45:35.960 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:45:35.977763+0000 2024-03-29T10:45:35.960 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:35.960 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:35.960 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:35.960 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:45:35.960 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:45:35.960 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:35.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.978+0000 7f544fd38640 1 Processor -- start 2024-03-29T10:45:35.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.978+0000 7f544fd38640 1 -- start start 2024-03-29T10:45:35.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.978+0000 7f544fd38640 10 monclient: init 2024-03-29T10:45:35.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.979+0000 7f544fd38640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:35.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.979+0000 7f544fd38640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:35.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.979+0000 7f544fd38640 10 monclient(hunting): picked mon.noname-a con 0x7f544806d9d0 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:35.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.979+0000 7f544fd38640 10 monclient(hunting): picked mon.noname-b con 0x7f544813e820 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:35.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.979+0000 7f544fd38640 10 monclient(hunting): picked mon.noname-c con 0x7f54480af1d0 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:35.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.979+0000 7f544fd38640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54480aed10 con 0x7f544806d9d0 2024-03-29T10:45:35.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.979+0000 7f544fd38640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54480afd50 con 0x7f54480af1d0 2024-03-29T10:45:35.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.979+0000 7f544fd38640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54480ab490 con 0x7f544813e820 2024-03-29T10:45:35.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.979+0000 7f544fd38640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:35.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.979+0000 7f544fd38640 10 monclient(hunting): authenticate will time out at 956.176723s 2024-03-29T10:45:35.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.982+0000 7f544dad4640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f544806d9d0 0x7f54480b05f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37341/0 (socket says 172.21.15.98:37341) 2024-03-29T10:45:35.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.982+0000 7f544dad4640 1 -- 172.21.15.98:0/2147551177 learned_addr learned my addr 172.21.15.98:0/2147551177 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:35.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.982+0000 7f544cad2640 1 -- 172.21.15.98:0/2147551177 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346126031 0 0) 0x7f54480aed10 con 0x7f544806d9d0 2024-03-29T10:45:35.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.982+0000 7f544cad2640 10 monclient(hunting): _negotiate 2024-03-29T10:45:35.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.982+0000 7f544cad2640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:35.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.982+0000 7f544cad2640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:35.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.982+0000 7f544cad2640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:45:35.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.982+0000 7f544cad2640 1 -- 172.21.15.98:0/2147551177 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5434003520 con 0x7f544806d9d0 2024-03-29T10:45:35.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.983+0000 7f544cad2640 1 -- 172.21.15.98:0/2147551177 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2353882268 0 0) 0x7f5434003520 con 0x7f544806d9d0 2024-03-29T10:45:35.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.983+0000 7f544cad2640 1 -- 172.21.15.98:0/2147551177 >> v1:172.21.15.98:6790/0 conn(0x7f544813e820 legacy=0x7f544813ec50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:35.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.984+0000 7f544cad2640 1 -- 172.21.15.98:0/2147551177 >> v1:172.21.15.189:6789/0 conn(0x7f54480af1d0 legacy=0x7f54480af640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:35.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 10 monclient: _finish_hunting 0 2024-03-29T10:45:35.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 1 monclient: found mon.noname-a 2024-03-29T10:45:35.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:35.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:45:35.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 1 -- 172.21.15.98:0/2147551177 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54480ab770 con 0x7f544806d9d0 2024-03-29T10:45:35.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 10 monclient: _finish_auth 0 2024-03-29T10:45:35.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 10 monclient: _check_auth_tickets 2024-03-29T10:45:35.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 1 -- 172.21.15.98:0/2147551177 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5440001a10 con 0x7f544806d9d0 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 10 monclient: dump: 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:35.970 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 1 -- 172.21.15.98:0/2147551177 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5440001c70 con 0x7f544806d9d0 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 1 -- 172.21.15.98:0/2147551177 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f54400020c0 con 0x7f544806d9d0 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.985+0000 7f544cad2640 10 monclient: dump: 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:35.971 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:35.972 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:35.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.986+0000 7f544fd38640 5 monclient: authenticate success, global_id 4257 2024-03-29T10:45:35.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.986+0000 7f544fd38640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:35.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.986+0000 7f544fd38640 4 set_mon_vals no callback set 2024-03-29T10:45:35.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.987+0000 7f544fd38640 10 monclient: shutdown 2024-03-29T10:45:35.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.987+0000 7f544fd38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:35.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.987+0000 7f544fd38640 1 -- 172.21.15.98:0/2147551177 >> v1:172.21.15.98:6789/0 conn(0x7f544806d9d0 legacy=0x7f54480b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:35.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.990+0000 7f544fd38640 1 -- 172.21.15.98:0/2147551177 shutdown_connections 2024-03-29T10:45:35.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.990+0000 7f544fd38640 1 -- 172.21.15.98:0/2147551177 >> 172.21.15.98:0/2147551177 conn(0x7f544806b660 msgr2=0x7f544806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:35.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.994+0000 7f544fd38640 1 -- 172.21.15.98:0/2147551177 shutdown_connections 2024-03-29T10:45:35.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.994+0000 7f544fd38640 1 -- 172.21.15.98:0/2147551177 wait complete. 2024-03-29T10:45:35.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.995+0000 7f544fd38640 10 monclient: build_initial_monmap 2024-03-29T10:45:35.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.995+0000 7f544fd38640 10 monclient: monmap: 2024-03-29T10:45:35.987 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:35.987 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:35.987 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:45:35.987 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:45:35.987 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:35.987 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.996+0000 7f544fd38640 1 librados: starting msgr at 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.996+0000 7f544fd38640 1 librados: starting objecter 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.996+0000 7f544fd38640 1 Processor -- start 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.996+0000 7f544fd38640 1 -- start start 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.996+0000 7f544fd38640 1 librados: setting wanted keys 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.996+0000 7f544fd38640 1 librados: calling monclient init 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.996+0000 7f544fd38640 10 monclient: init 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.997+0000 7f544fd38640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.997+0000 7f544fd38640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.997+0000 7f544fd38640 10 monclient(hunting): picked mon.noname-a con 0x7f544806d9d0 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:35.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.997+0000 7f544fd38640 10 monclient(hunting): picked mon.noname-c con 0x7f54480af1d0 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:35.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.997+0000 7f544fd38640 10 monclient(hunting): picked mon.noname-b con 0x7f544813e820 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:35.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.997+0000 7f544fd38640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5448070db0 con 0x7f544806d9d0 2024-03-29T10:45:35.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.997+0000 7f544fd38640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54481383f0 con 0x7f544813e820 2024-03-29T10:45:35.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.997+0000 7f544fd38640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f544806e510 con 0x7f54480af1d0 2024-03-29T10:45:35.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.997+0000 7f544fd38640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:35.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:35.997+0000 7f544fd38640 10 monclient(hunting): authenticate will time out at 956.194120s 2024-03-29T10:45:35.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.003+0000 7f544dad4640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f544806d9d0 0x7f54480706a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37463/0 (socket says 172.21.15.98:37463) 2024-03-29T10:45:35.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.003+0000 7f544dad4640 1 -- 172.21.15.98:0/1585209250 learned_addr learned my addr 172.21.15.98:0/1585209250 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:35.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.003+0000 7f543e7fc640 10 client.?.objecter ms_handle_connect 0x7f544806d9d0 2024-03-29T10:45:35.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.003+0000 7f543e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:35.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.003+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657786563 0 0) 0x7f5448070db0 con 0x7f544806d9d0 2024-03-29T10:45:35.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.003+0000 7f543e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:35.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.003+0000 7f543e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:35.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.003+0000 7f543e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:35.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.003+0000 7f543e7fc640 10 monclient(hunting): my global_id is 4260 2024-03-29T10:45:35.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.004+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542c004eb0 con 0x7f544806d9d0 2024-03-29T10:45:35.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.004+0000 7f543e7fc640 10 client.?.objecter ms_handle_connect 0x7f544813e820 2024-03-29T10:45:35.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.004+0000 7f543e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:35.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.007+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757551774 0 0) 0x7f54481383f0 con 0x7f544813e820 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.007+0000 7f543e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.007+0000 7f543e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.007+0000 7f543e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.007+0000 7f543e7fc640 10 monclient(hunting): my global_id is 4264 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.007+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5448070db0 con 0x7f544813e820 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.007+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 583792851 0 0) 0x7f542c004eb0 con 0x7f544806d9d0 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.007+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54481383f0 con 0x7f544806d9d0 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.007+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f54400019d0 con 0x7f544806d9d0 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.007+0000 7f543e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.007+0000 7f543e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.007+0000 7f543e7fc640 10 monclient(hunting): dump: 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:35.996 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 569511875 0 0) 0x7f5448070db0 con 0x7f544813e820 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f542c004eb0 con 0x7f544813e820 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5438001730 con 0x7f544813e820 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 10 monclient(hunting): dump: 2024-03-29T10:45:35.997 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 767150941 0 0) 0x7f542c004eb0 con 0x7f544813e820 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 >> v1:172.21.15.98:6790/0 conn(0x7f54480af1d0 legacy=0x7f54481375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 >> v1:172.21.15.98:6789/0 conn(0x7f544806d9d0 legacy=0x7f54480706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:35.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 1 monclient: found mon.b 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544806efd0 con 0x7f544813e820 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f543e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f544fd38640 5 monclient: authenticate success, global_id 4264 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f544fd38640 10 monclient: _renew_subs 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f544fd38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f544fd38640 1 -- 172.21.15.98:0/1585209250 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5448134f90 con 0x7f544813e820 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f544fd38640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f544fd38640 10 monclient: _renew_subs 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f544fd38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:35.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f544fd38640 1 -- 172.21.15.98:0/1585209250 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f544806ddb0 con 0x7f544813e820 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.008+0000 7f544fd38640 1 librados: init done 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.009+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5438001c40 con 0x7f544813e820 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.009+0000 7f543e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.009+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5438003830 con 0x7f544813e820 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.009+0000 7f543e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.009+0000 7f543e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.009+0000 7f543e7fc640 10 monclient: dump: 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:36.000 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.009+0000 7f543ffff640 4 set_mon_vals no callback set 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.010+0000 7f542bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.011+0000 7f542bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.011+0000 7f542bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.011+0000 7f542bfff640 1 -- 172.21.15.98:0/1585209250 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544806e6a0 con 0x7f544813e820 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.011+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f5438019800 con 0x7f544813e820 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.012+0000 7f544dad4640 10 monclient: get_auth_request con 0x7f542c06de80 auth_method 0 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.012+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f54380836e0 con 0x7f544813e820 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.012+0000 7f543e7fc640 10 client.4264.objecter ms_dispatch 0x7f5448000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.012+0000 7f543e7fc640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.012+0000 7f543e7fc640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:36.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.012+0000 7f543e7fc640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-29T10:45:36.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.012+0000 7f543e7fc640 10 client.4264.objecter ms_handle_connect 0x7f542c06de80 2024-03-29T10:45:36.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.019+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f543804efc0 con 0x7f544813e820 2024-03-29T10:45:36.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.019+0000 7f543e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:36.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.019+0000 7f543e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:36.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.341+0000 7f542bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:36.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.341+0000 7f542bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:36.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.341+0000 7f542bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:36.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.341+0000 7f542bfff640 1 -- 172.21.15.98:0/1585209250 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5448075f10 con 0x7f544813e820 2024-03-29T10:45:36.325 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.342+0000 7f543e7fc640 1 -- 172.21.15.98:0/1585209250 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f5438050ef0 con 0x7f544813e820 2024-03-29T10:45:36.325 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.342+0000 7f543e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:36.325 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.342+0000 7f543e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:36.329 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:45:36.329 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:36.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.348+0000 7f542bfff640 10 librados: watch_flush enter 2024-03-29T10:45:36.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.351+0000 7f542bfff640 10 librados: watch_flush exit 2024-03-29T10:45:36.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.351+0000 7f542bfff640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-29T10:45:36.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.351+0000 7f542bfff640 10 client.4264.objecter successfully canceled tick 2024-03-29T10:45:36.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.351+0000 7f542bfff640 1 -- 172.21.15.98:0/1585209250 >> v1:172.21.15.98:6816/1481799426 conn(0x7f542c06de80 legacy=0x7f542c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:36.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.351+0000 7f542bfff640 10 monclient: shutdown 2024-03-29T10:45:36.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.351+0000 7f542bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:36.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.351+0000 7f542bfff640 1 -- 172.21.15.98:0/1585209250 >> v1:172.21.15.189:6789/0 conn(0x7f544813e820 legacy=0x7f544812b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:36.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.351+0000 7f542bfff640 1 -- 172.21.15.98:0/1585209250 shutdown_connections 2024-03-29T10:45:36.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.351+0000 7f542bfff640 1 -- 172.21.15.98:0/1585209250 >> 172.21.15.98:0/1585209250 conn(0x7f544806b660 msgr2=0x7f544814d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:36.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.352+0000 7f542bfff640 1 -- 172.21.15.98:0/1585209250 shutdown_connections 2024-03-29T10:45:36.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.352+0000 7f542bfff640 1 -- 172.21.15.98:0/1585209250 wait complete. 2024-03-29T10:45:36.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:36.352+0000 7f542bfff640 1 librados: shutdown 2024-03-29T10:45:36.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:43.161 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:43.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.345+0000 7f4f9a582640 10 monclient: get_monmap_and_config 2024-03-29T10:45:43.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.345+0000 7f4f9a582640 10 monclient: build_initial_monmap 2024-03-29T10:45:43.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.345+0000 7f4f9a582640 10 monclient: monmap: 2024-03-29T10:45:43.330 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:43.330 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:43.330 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:45:43.346576+0000 2024-03-29T10:45:43.330 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:45:43.346576+0000 2024-03-29T10:45:43.330 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:43.330 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:43.330 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:43.330 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:45:43.330 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:45:43.330 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:43.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.347+0000 7f4f9a582640 1 Processor -- start 2024-03-29T10:45:43.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.347+0000 7f4f9a582640 1 -- start start 2024-03-29T10:45:43.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.347+0000 7f4f9a582640 10 monclient: init 2024-03-29T10:45:43.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.348+0000 7f4f9a582640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:43.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.348+0000 7f4f9a582640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:43.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.348+0000 7f4f9a582640 10 monclient(hunting): picked mon.noname-a con 0x7f4f940a9ac0 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:43.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.348+0000 7f4f9a582640 10 monclient(hunting): picked mon.noname-b con 0x7f4f9413ee90 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:43.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.348+0000 7f4f9a582640 10 monclient(hunting): picked mon.noname-c con 0x7f4f940af900 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:43.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.348+0000 7f4f9a582640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f940afd10 con 0x7f4f940a9ac0 2024-03-29T10:45:43.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.348+0000 7f4f9a582640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f940af6f0 con 0x7f4f940af900 2024-03-29T10:45:43.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.348+0000 7f4f9a582640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f940aac80 con 0x7f4f9413ee90 2024-03-29T10:45:43.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.348+0000 7f4f9a582640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:43.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.348+0000 7f4f9a582640 10 monclient(hunting): authenticate will time out at 963.545612s 2024-03-29T10:45:43.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.351+0000 7f4f98b1f640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4f940af900 0x7f4f94149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:51247/0 (socket says 172.21.15.98:51247) 2024-03-29T10:45:43.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.351+0000 7f4f98b1f640 1 -- 172.21.15.98:0/2888508420 learned_addr learned my addr 172.21.15.98:0/2888508420 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:43.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.351+0000 7f4f92ffd640 1 -- 172.21.15.98:0/2888508420 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460658670 0 0) 0x7f4f940af6f0 con 0x7f4f940af900 2024-03-29T10:45:43.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.351+0000 7f4f92ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:43.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.351+0000 7f4f92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:43.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.351+0000 7f4f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:43.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.351+0000 7f4f92ffd640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:45:43.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.351+0000 7f4f92ffd640 1 -- 172.21.15.98:0/2888508420 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f7c003500 con 0x7f4f940af900 2024-03-29T10:45:43.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.352+0000 7f4f92ffd640 1 -- 172.21.15.98:0/2888508420 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3681999216 0 0) 0x7f4f7c003500 con 0x7f4f940af900 2024-03-29T10:45:43.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.352+0000 7f4f92ffd640 1 -- 172.21.15.98:0/2888508420 >> v1:172.21.15.98:6790/0 conn(0x7f4f9413ee90 legacy=0x7f4f9413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:43.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f92ffd640 1 -- 172.21.15.98:0/2888508420 >> v1:172.21.15.98:6789/0 conn(0x7f4f940a9ac0 legacy=0x7f4f940adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:43.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f92ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:43.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f92ffd640 1 monclient: found mon.noname-c 2024-03-29T10:45:43.338 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f92ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f92ffd640 1 -- 172.21.15.98:0/2888508420 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f940aaf60 con 0x7f4f940af900 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f92ffd640 10 monclient: _finish_auth 0 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f92ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f92ffd640 1 -- 172.21.15.98:0/2888508420 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4f8c005030 con 0x7f4f940af900 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f92ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f92ffd640 10 monclient: dump: 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:43.339 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:43.340 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:43.340 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:43.340 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:43.340 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:43.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f9a582640 5 monclient: authenticate success, global_id 4267 2024-03-29T10:45:43.340 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.354+0000 7f4f9a582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.357+0000 7f4f92ffd640 1 -- 172.21.15.98:0/2888508420 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f8c005290 con 0x7f4f940af900 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.357+0000 7f4f92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.357+0000 7f4f92ffd640 1 -- 172.21.15.98:0/2888508420 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4f8c001830 con 0x7f4f940af900 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.357+0000 7f4f92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.357+0000 7f4f92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.357+0000 7f4f92ffd640 10 monclient: dump: 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:43.341 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:43.342 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:43.342 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:43.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.358+0000 7f4f9a582640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:43.342 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.358+0000 7f4f9a582640 4 set_mon_vals no callback set 2024-03-29T10:45:43.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.359+0000 7f4f9a582640 10 monclient: shutdown 2024-03-29T10:45:43.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.359+0000 7f4f9a582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:43.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.359+0000 7f4f9a582640 1 -- 172.21.15.98:0/2888508420 >> v1:172.21.15.189:6789/0 conn(0x7f4f940af900 legacy=0x7f4f94149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:43.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.359+0000 7f4f9a582640 1 -- 172.21.15.98:0/2888508420 shutdown_connections 2024-03-29T10:45:43.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.359+0000 7f4f9a582640 1 -- 172.21.15.98:0/2888508420 >> 172.21.15.98:0/2888508420 conn(0x7f4f940b35e0 msgr2=0x7f4f940b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:43.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.359+0000 7f4f9a582640 1 -- 172.21.15.98:0/2888508420 shutdown_connections 2024-03-29T10:45:43.343 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.359+0000 7f4f9a582640 1 -- 172.21.15.98:0/2888508420 wait complete. 2024-03-29T10:45:43.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.364+0000 7f4f9a582640 10 monclient: build_initial_monmap 2024-03-29T10:45:43.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.364+0000 7f4f9a582640 10 monclient: monmap: 2024-03-29T10:45:43.349 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.364+0000 7f4f9a582640 1 librados: starting msgr at 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.364+0000 7f4f9a582640 1 librados: starting objecter 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.364+0000 7f4f9a582640 1 Processor -- start 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.365+0000 7f4f9a582640 1 -- start start 2024-03-29T10:45:43.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.365+0000 7f4f9a582640 1 librados: setting wanted keys 2024-03-29T10:45:43.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.365+0000 7f4f9a582640 1 librados: calling monclient init 2024-03-29T10:45:43.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.365+0000 7f4f9a582640 10 monclient: init 2024-03-29T10:45:43.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.366+0000 7f4f9a582640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:43.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.366+0000 7f4f9a582640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:43.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.366+0000 7f4f9a582640 10 monclient(hunting): picked mon.noname-c con 0x7f4f940a9ac0 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:43.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.366+0000 7f4f9a582640 10 monclient(hunting): picked mon.noname-b con 0x7f4f940af900 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:43.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.366+0000 7f4f9a582640 10 monclient(hunting): picked mon.noname-a con 0x7f4f9413ee90 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:43.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.366+0000 7f4f9a582640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f94142260 con 0x7f4f9413ee90 2024-03-29T10:45:43.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.366+0000 7f4f9a582640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9413e250 con 0x7f4f940af900 2024-03-29T10:45:43.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.366+0000 7f4f9a582640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9413e3e0 con 0x7f4f940a9ac0 2024-03-29T10:45:43.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.366+0000 7f4f9a582640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:43.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.366+0000 7f4f9a582640 10 monclient(hunting): authenticate will time out at 963.563407s 2024-03-29T10:45:43.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.377+0000 7f4f937fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4f940af900 0x7f4f9413f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:51377/0 (socket says 172.21.15.98:51377) 2024-03-29T10:45:43.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.377+0000 7f4f937fe640 1 -- 172.21.15.98:0/1720377368 learned_addr learned my addr 172.21.15.98:0/1720377368 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:43.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.377+0000 7f4f90ff9640 10 client.?.objecter ms_handle_connect 0x7f4f940af900 2024-03-29T10:45:43.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.377+0000 7f4f90ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:43.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.377+0000 7f4f90ff9640 10 client.?.objecter ms_handle_connect 0x7f4f9413ee90 2024-03-29T10:45:43.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.377+0000 7f4f90ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:43.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.377+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018783275 0 0) 0x7f4f9413e250 con 0x7f4f940af900 2024-03-29T10:45:43.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.377+0000 7f4f90ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:45:43.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.377+0000 7f4f90ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:43.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.377+0000 7f4f90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:43.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.377+0000 7f4f90ff9640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:45:43.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.377+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f80004eb0 con 0x7f4f940af900 2024-03-29T10:45:43.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089845781 0 0) 0x7f4f94142260 con 0x7f4f9413ee90 2024-03-29T10:45:43.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:45:43.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:43.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:43.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 10 monclient(hunting): my global_id is 4263 2024-03-29T10:45:43.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f9413e250 con 0x7f4f9413ee90 2024-03-29T10:45:43.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 206731838 0 0) 0x7f4f80004eb0 con 0x7f4f940af900 2024-03-29T10:45:43.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f94142260 con 0x7f4f940af900 2024-03-29T10:45:43.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4f88001960 con 0x7f4f940af900 2024-03-29T10:45:43.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:43.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:43.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 10 monclient(hunting): dump: 2024-03-29T10:45:43.364 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 10 client.?.objecter ms_handle_connect 0x7f4f940a9ac0 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.378+0000 7f4f90ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 360531281 0 0) 0x7f4f94142260 con 0x7f4f940af900 2024-03-29T10:45:43.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 >> v1:172.21.15.98:6790/0 conn(0x7f4f940a9ac0 legacy=0x7f4f94136610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:43.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 >> v1:172.21.15.98:6789/0 conn(0x7f4f9413ee90 legacy=0x7f4f9413fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:43.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f90ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:45:43.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f90ff9640 1 monclient: found mon.b 2024-03-29T10:45:43.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f90ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:43.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f90ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:43.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f94141df0 con 0x7f4f940af900 2024-03-29T10:45:43.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f90ff9640 10 monclient: _finish_auth 0 2024-03-29T10:45:43.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f90ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:45:43.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:43.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f9a582640 5 monclient: authenticate success, global_id 4270 2024-03-29T10:45:43.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f9a582640 10 monclient: _renew_subs 2024-03-29T10:45:43.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f9a582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:43.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f9a582640 1 -- 172.21.15.98:0/1720377368 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9413de00 con 0x7f4f940af900 2024-03-29T10:45:43.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f9a582640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:43.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f9a582640 10 monclient: _renew_subs 2024-03-29T10:45:43.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f9a582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:43.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f9a582640 1 -- 172.21.15.98:0/1720377368 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f94136f00 con 0x7f4f940af900 2024-03-29T10:45:43.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.379+0000 7f4f9a582640 1 librados: init done 2024-03-29T10:45:43.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.382+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f88001ac0 con 0x7f4f940af900 2024-03-29T10:45:43.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.382+0000 7f4f90ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:43.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.382+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4f880038a0 con 0x7f4f940af900 2024-03-29T10:45:43.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.382+0000 7f4f90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:43.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.382+0000 7f4f90ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:43.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.382+0000 7f4f90ff9640 10 monclient: dump: 2024-03-29T10:45:43.368 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:43.368 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:43.368 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:43.368 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:43.368 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:43.369 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:43.369 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:43.369 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:43.369 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:43.369 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:43.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.383+0000 7f4f9a582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:43.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.383+0000 7f4f9a582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:43.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.383+0000 7f4f9a582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:43.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.383+0000 7f4f9a582640 1 -- 172.21.15.98:0/1720377368 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f941423f0 con 0x7f4f940af900 2024-03-29T10:45:43.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.383+0000 7f4f927fc640 4 set_mon_vals no callback set 2024-03-29T10:45:43.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.383+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f4f88003ac0 con 0x7f4f940af900 2024-03-29T10:45:43.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.384+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f4f88084a60 con 0x7f4f940af900 2024-03-29T10:45:43.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.384+0000 7f4f90ff9640 10 client.4270.objecter ms_dispatch 0x7f4f94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:43.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.384+0000 7f4f90ff9640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:43.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.384+0000 7f4f90ff9640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:43.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.384+0000 7f4f90ff9640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T10:45:43.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.384+0000 7f4f93fff640 10 monclient: get_auth_request con 0x7f4f8006dd30 auth_method 0 2024-03-29T10:45:43.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.384+0000 7f4f90ff9640 10 client.4270.objecter ms_handle_connect 0x7f4f8006dd30 2024-03-29T10:45:43.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.391+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f880504b0 con 0x7f4f940af900 2024-03-29T10:45:43.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.391+0000 7f4f90ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:43.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.391+0000 7f4f90ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:43.735 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.752+0000 7f4f9a582640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:43.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.752+0000 7f4f9a582640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:43.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.752+0000 7f4f9a582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:43.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.752+0000 7f4f9a582640 1 -- 172.21.15.98:0/1720377368 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f94075f10 con 0x7f4f940af900 2024-03-29T10:45:43.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.753+0000 7f4f90ff9640 1 -- 172.21.15.98:0/1720377368 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f4f880523e0 con 0x7f4f940af900 2024-03-29T10:45:43.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.753+0000 7f4f90ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:43.736 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.753+0000 7f4f90ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:43.736 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:45:43.736 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:43.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.756+0000 7f4f9a582640 10 librados: watch_flush enter 2024-03-29T10:45:43.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.756+0000 7f4f9a582640 10 librados: watch_flush exit 2024-03-29T10:45:43.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.756+0000 7f4f9a582640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T10:45:43.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.756+0000 7f4f9a582640 10 client.4270.objecter successfully canceled tick 2024-03-29T10:45:43.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.756+0000 7f4f9a582640 1 -- 172.21.15.98:0/1720377368 >> v1:172.21.15.98:6816/1481799426 conn(0x7f4f8006dd30 legacy=0x7f4f800701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:43.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.757+0000 7f4f9a582640 10 monclient: shutdown 2024-03-29T10:45:43.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.757+0000 7f4f9a582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:43.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.757+0000 7f4f9a582640 1 -- 172.21.15.98:0/1720377368 >> v1:172.21.15.189:6789/0 conn(0x7f4f940af900 legacy=0x7f4f9413f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:43.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.757+0000 7f4f9a582640 1 -- 172.21.15.98:0/1720377368 shutdown_connections 2024-03-29T10:45:43.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.757+0000 7f4f9a582640 1 -- 172.21.15.98:0/1720377368 >> 172.21.15.98:0/1720377368 conn(0x7f4f940b35e0 msgr2=0x7f4f9413b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:43.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.757+0000 7f4f9a582640 1 -- 172.21.15.98:0/1720377368 shutdown_connections 2024-03-29T10:45:43.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.757+0000 7f4f9a582640 1 -- 172.21.15.98:0/1720377368 wait complete. 2024-03-29T10:45:43.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:43.757+0000 7f4f9a582640 1 librados: shutdown 2024-03-29T10:45:43.756 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:50.564 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:50.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.739+0000 7f879f93b640 10 monclient: get_monmap_and_config 2024-03-29T10:45:50.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.739+0000 7f879f93b640 10 monclient: build_initial_monmap 2024-03-29T10:45:50.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.739+0000 7f879f93b640 10 monclient: monmap: 2024-03-29T10:45:50.722 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:50.722 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:50.722 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:45:50.740575+0000 2024-03-29T10:45:50.722 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:45:50.740575+0000 2024-03-29T10:45:50.722 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:50.723 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:50.723 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:50.723 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:45:50.723 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:45:50.723 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:50.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.740+0000 7f879f93b640 1 Processor -- start 2024-03-29T10:45:50.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.741+0000 7f879f93b640 1 -- start start 2024-03-29T10:45:50.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.741+0000 7f879f93b640 10 monclient: init 2024-03-29T10:45:50.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.742+0000 7f879f93b640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:50.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.742+0000 7f879f93b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:45:50.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.742+0000 7f879f93b640 10 monclient(hunting): picked mon.noname-c con 0x7f87981457f0 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:50.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.742+0000 7f879f93b640 10 monclient(hunting): picked mon.noname-b con 0x7f879813f650 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:50.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.742+0000 7f879f93b640 10 monclient(hunting): picked mon.noname-a con 0x7f87980a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:50.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.742+0000 7f879f93b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87980aaef0 con 0x7f87980a9d70 2024-03-29T10:45:50.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.742+0000 7f879f93b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8798145bd0 con 0x7f87981457f0 2024-03-29T10:45:50.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.742+0000 7f879f93b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8798145eb0 con 0x7f879813f650 2024-03-29T10:45:50.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.742+0000 7f879f93b640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:50.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.742+0000 7f879f93b640 10 monclient(hunting): authenticate will time out at 970.939260s 2024-03-29T10:45:50.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.742+0000 7f879ded8640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f87980a9d70 0x7f879814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:42136/0 (socket says 172.21.15.98:42136) 2024-03-29T10:45:50.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.742+0000 7f879ded8640 1 -- 172.21.15.98:0/3179621230 learned_addr learned my addr 172.21.15.98:0/3179621230 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:50.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.743+0000 7f878ffff640 1 -- 172.21.15.98:0/3179621230 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117739625 0 0) 0x7f87980aaef0 con 0x7f87980a9d70 2024-03-29T10:45:50.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.743+0000 7f878ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:50.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.743+0000 7f878ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:50.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.743+0000 7f878ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:50.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.743+0000 7f878ffff640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:45:50.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.743+0000 7f878ffff640 1 -- 172.21.15.98:0/3179621230 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8784003520 con 0x7f87980a9d70 2024-03-29T10:45:50.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.743+0000 7f878ffff640 1 -- 172.21.15.98:0/3179621230 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231070931 0 0) 0x7f8798145bd0 con 0x7f87981457f0 2024-03-29T10:45:50.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.743+0000 7f878ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:50.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.743+0000 7f878ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:50.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.743+0000 7f878ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:50.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.743+0000 7f878ffff640 10 monclient(hunting): my global_id is 4273 2024-03-29T10:45:50.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.743+0000 7f878ffff640 1 -- 172.21.15.98:0/3179621230 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87980aaef0 con 0x7f87981457f0 2024-03-29T10:45:50.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 1 -- 172.21.15.98:0/3179621230 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1314419605 0 0) 0x7f87980aaef0 con 0x7f87981457f0 2024-03-29T10:45:50.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 1 -- 172.21.15.98:0/3179621230 >> v1:172.21.15.98:6790/0 conn(0x7f879813f650 legacy=0x7f87980b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:50.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 1 -- 172.21.15.98:0/3179621230 >> v1:172.21.15.98:6789/0 conn(0x7f87980a9d70 legacy=0x7f879814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 1 monclient: found mon.noname-c 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 1 -- 172.21.15.98:0/3179621230 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8798146340 con 0x7f87981457f0 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 10 monclient: _finish_auth 0 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 1 -- 172.21.15.98:0/3179621230 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8794001670 con 0x7f87981457f0 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f878ffff640 10 monclient: dump: 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:50.731 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f879f93b640 5 monclient: authenticate success, global_id 4273 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.744+0000 7f879f93b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.745+0000 7f878ffff640 1 -- 172.21.15.98:0/3179621230 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87940018c0 con 0x7f87981457f0 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.745+0000 7f878ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.745+0000 7f878ffff640 1 -- 172.21.15.98:0/3179621230 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8794001c50 con 0x7f87981457f0 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.745+0000 7f878ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.745+0000 7f878ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:50.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.745+0000 7f878ffff640 10 monclient: dump: 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.745+0000 7f879f93b640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.745+0000 7f879f93b640 4 set_mon_vals no callback set 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.746+0000 7f879f93b640 10 monclient: shutdown 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.746+0000 7f879f93b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.746+0000 7f879f93b640 1 -- 172.21.15.98:0/3179621230 >> v1:172.21.15.189:6789/0 conn(0x7f87981457f0 legacy=0x7f879813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:50.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.746+0000 7f879f93b640 1 -- 172.21.15.98:0/3179621230 shutdown_connections 2024-03-29T10:45:50.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.746+0000 7f879f93b640 1 -- 172.21.15.98:0/3179621230 >> 172.21.15.98:0/3179621230 conn(0x7f8798066a00 msgr2=0x7f8798066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:50.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.747+0000 7f879f93b640 1 -- 172.21.15.98:0/3179621230 shutdown_connections 2024-03-29T10:45:50.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.747+0000 7f879f93b640 1 -- 172.21.15.98:0/3179621230 wait complete. 2024-03-29T10:45:50.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.748+0000 7f879f93b640 10 monclient: build_initial_monmap 2024-03-29T10:45:50.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.748+0000 7f879f93b640 10 monclient: monmap: 2024-03-29T10:45:50.741 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:50.741 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:50.741 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:45:50.742 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:45:50.742 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:50.742 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:50.742 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:50.742 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:45:50.742 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:45:50.742 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:50.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.748+0000 7f879f93b640 1 librados: starting msgr at 2024-03-29T10:45:50.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.748+0000 7f879f93b640 1 librados: starting objecter 2024-03-29T10:45:50.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.748+0000 7f879f93b640 1 Processor -- start 2024-03-29T10:45:50.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.748+0000 7f879f93b640 1 -- start start 2024-03-29T10:45:50.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.748+0000 7f879f93b640 1 librados: setting wanted keys 2024-03-29T10:45:50.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.748+0000 7f879f93b640 1 librados: calling monclient init 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.748+0000 7f879f93b640 10 monclient: init 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.749+0000 7f879f93b640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.749+0000 7f879f93b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f879f93b640 10 monclient(hunting): picked mon.noname-c con 0x7f87980a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f879f93b640 10 monclient(hunting): picked mon.noname-b con 0x7f879813f650 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f879f93b640 10 monclient(hunting): picked mon.noname-a con 0x7f87981457f0 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f879f93b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f879806e100 con 0x7f87981457f0 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f879f93b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8798142480 con 0x7f879813f650 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f879f93b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8798142c80 con 0x7f87980a9d70 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f879f93b640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f879f93b640 10 monclient(hunting): authenticate will time out at 970.947224s 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f879d6d7640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f87980a9d70 0x7f8798141d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:47868/0 (socket says 172.21.15.98:47868) 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f879d6d7640 1 -- 172.21.15.98:0/2717516412 learned_addr learned my addr 172.21.15.98:0/2717516412 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f878dffb640 10 client.?.objecter ms_handle_connect 0x7f879813f650 2024-03-29T10:45:50.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f878dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f878dffb640 10 client.?.objecter ms_handle_connect 0x7f87981457f0 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f878dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f878dffb640 10 client.?.objecter ms_handle_connect 0x7f87980a9d70 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.750+0000 7f878dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276693721 0 0) 0x7f8798142480 con 0x7f879813f650 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f877c004eb0 con 0x7f879813f650 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787506257 0 0) 0x7f8798142c80 con 0x7f87980a9d70 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:45:50.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8798142480 con 0x7f87980a9d70 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960310077 0 0) 0x7f879806e100 con 0x7f87981457f0 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): my global_id is 4269 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8798142c80 con 0x7f87981457f0 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1588672494 0 0) 0x7f877c004eb0 con 0x7f879813f650 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f879806e100 con 0x7f879813f650 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8788001930 con 0x7f879813f650 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): dump: 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:50.745 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 391039384 0 0) 0x7f8798142c80 con 0x7f87981457f0 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f877c004eb0 con 0x7f87981457f0 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8790001cd0 con 0x7f87981457f0 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:50.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.751+0000 7f878dffb640 10 monclient(hunting): dump: 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3442425717 0 0) 0x7f8798142480 con 0x7f87980a9d70 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8798142c80 con 0x7f87980a9d70 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1735116406 0 0) 0x7f879806e100 con 0x7f879813f650 2024-03-29T10:45:50.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 >> v1:172.21.15.98:6790/0 conn(0x7f87980a9d70 legacy=0x7f8798141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 >> v1:172.21.15.98:6789/0 conn(0x7f87981457f0 legacy=0x7f8798137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f878dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f878dffb640 1 monclient: found mon.b 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f878dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f878dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87981426d0 con 0x7f879813f650 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f878dffb640 10 monclient: _finish_auth 0 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f878dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f878dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f879f93b640 5 monclient: authenticate success, global_id 4276 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f879f93b640 10 monclient: _renew_subs 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f879f93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f879f93b640 1 -- 172.21.15.98:0/2717516412 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87981429a0 con 0x7f879813f650 2024-03-29T10:45:50.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f879f93b640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f879f93b640 10 monclient: _renew_subs 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f879f93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f879f93b640 1 -- 172.21.15.98:0/2717516412 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f879813b570 con 0x7f879813f650 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.752+0000 7f879f93b640 1 librados: init done 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.753+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87880035f0 con 0x7f879813f650 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.753+0000 7f878dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.753+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8788003980 con 0x7f879813f650 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.753+0000 7f878dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.753+0000 7f878dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.753+0000 7f878dffb640 10 monclient: dump: 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:50.749 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:50.750 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:50.750 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:50.750 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:50.750 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:50.750 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:50.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.753+0000 7f878f7fe640 4 set_mon_vals no callback set 2024-03-29T10:45:50.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.754+0000 7f879f93b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:50.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.754+0000 7f879f93b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:50.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.754+0000 7f879f93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:50.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.754+0000 7f879f93b640 1 -- 172.21.15.98:0/2717516412 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8778005080 con 0x7f879813f650 2024-03-29T10:45:50.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.755+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f8788003ae0 con 0x7f879813f650 2024-03-29T10:45:50.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.756+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f8788083840 con 0x7f879813f650 2024-03-29T10:45:50.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.756+0000 7f878dffb640 10 client.4276.objecter ms_dispatch 0x7f8798000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:50.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.756+0000 7f878dffb640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:50.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.756+0000 7f878dffb640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:50.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.756+0000 7f878dffb640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-29T10:45:50.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.756+0000 7f879d6d7640 10 monclient: get_auth_request con 0x7f877c06e180 auth_method 0 2024-03-29T10:45:50.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.756+0000 7f878dffb640 10 client.4276.objecter ms_handle_connect 0x7f877c06e180 2024-03-29T10:45:50.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.763+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f878804f180 con 0x7f879813f650 2024-03-29T10:45:50.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.763+0000 7f878dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:50.756 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:50.763+0000 7f878dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:51.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.158+0000 7f879f93b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:51.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.159+0000 7f879f93b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:51.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.159+0000 7f879f93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:45:51.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.159+0000 7f879f93b640 1 -- 172.21.15.98:0/2717516412 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8778005330 con 0x7f879813f650 2024-03-29T10:45:51.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.160+0000 7f878dffb640 1 -- 172.21.15.98:0/2717516412 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f8788082ef0 con 0x7f879813f650 2024-03-29T10:45:51.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.160+0000 7f878dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:51.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.160+0000 7f878dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:51.143 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:45:51.143 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:51.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.163+0000 7f879f93b640 10 librados: watch_flush enter 2024-03-29T10:45:51.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.163+0000 7f879f93b640 10 librados: watch_flush exit 2024-03-29T10:45:51.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.163+0000 7f879f93b640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-29T10:45:51.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.163+0000 7f879f93b640 10 client.4276.objecter successfully canceled tick 2024-03-29T10:45:51.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.163+0000 7f879f93b640 1 -- 172.21.15.98:0/2717516412 >> v1:172.21.15.98:6816/1481799426 conn(0x7f877c06e180 legacy=0x7f877c070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:51.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.163+0000 7f879f93b640 10 monclient: shutdown 2024-03-29T10:45:51.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.163+0000 7f879f93b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:51.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.163+0000 7f879f93b640 1 -- 172.21.15.98:0/2717516412 >> v1:172.21.15.189:6789/0 conn(0x7f879813f650 legacy=0x7f8798137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:51.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.164+0000 7f879f93b640 1 -- 172.21.15.98:0/2717516412 shutdown_connections 2024-03-29T10:45:51.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.164+0000 7f879f93b640 1 -- 172.21.15.98:0/2717516412 >> 172.21.15.98:0/2717516412 conn(0x7f8798066a00 msgr2=0x7f87980b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:51.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.164+0000 7f879f93b640 1 -- 172.21.15.98:0/2717516412 shutdown_connections 2024-03-29T10:45:51.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.164+0000 7f879f93b640 1 -- 172.21.15.98:0/2717516412 wait complete. 2024-03-29T10:45:51.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:51.164+0000 7f879f93b640 1 librados: shutdown 2024-03-29T10:45:51.164 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:57.971 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:58.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.121+0000 7fb9eaf89640 10 monclient: get_monmap_and_config 2024-03-29T10:45:58.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.121+0000 7fb9eaf89640 10 monclient: build_initial_monmap 2024-03-29T10:45:58.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.121+0000 7fb9eaf89640 10 monclient: monmap: 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:45:58.122253+0000 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:45:58.122253+0000 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.122+0000 7fb9eaf89640 1 Processor -- start 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.122+0000 7fb9eaf89640 1 -- start start 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.122+0000 7fb9eaf89640 10 monclient: init 2024-03-29T10:45:58.107 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.123+0000 7fb9eaf89640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.123+0000 7fb9eaf89640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.123+0000 7fb9eaf89640 10 monclient(hunting): picked mon.noname-b con 0x7fb9e40b0770 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.123+0000 7fb9eaf89640 10 monclient(hunting): picked mon.noname-a con 0x7fb9e414a700 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.123+0000 7fb9eaf89640 10 monclient(hunting): picked mon.noname-c con 0x7fb9e40ac5a0 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.123+0000 7fb9eaf89640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e40aba70 con 0x7fb9e414a700 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.123+0000 7fb9eaf89640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e40aecb0 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.123+0000 7fb9eaf89640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e40af150 con 0x7fb9e40b0770 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.123+0000 7fb9eaf89640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.123+0000 7fb9eaf89640 10 monclient(hunting): authenticate will time out at 978.320563s 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.123+0000 7fb9e8d25640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fb9e40b0770 0x7fb9e4149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:35376/0 (socket says 172.21.15.98:35376) 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.123+0000 7fb9e8d25640 1 -- 172.21.15.98:0/2627439210 learned_addr learned my addr 172.21.15.98:0/2627439210 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.124+0000 7fb9e37fe640 1 -- 172.21.15.98:0/2627439210 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527226165 0 0) 0x7fb9e40af150 con 0x7fb9e40b0770 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.124+0000 7fb9e37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.124+0000 7fb9e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:58.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.124+0000 7fb9e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:58.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.124+0000 7fb9e37fe640 10 monclient(hunting): my global_id is 4271 2024-03-29T10:45:58.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.124+0000 7fb9e37fe640 1 -- 172.21.15.98:0/2627439210 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc003520 con 0x7fb9e40b0770 2024-03-29T10:45:58.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.124+0000 7fb9e37fe640 1 -- 172.21.15.98:0/2627439210 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1446712312 0 0) 0x7fb9cc003520 con 0x7fb9e40b0770 2024-03-29T10:45:58.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.124+0000 7fb9e37fe640 1 -- 172.21.15.98:0/2627439210 >> v1:172.21.15.189:6789/0 conn(0x7fb9e40ac5a0 legacy=0x7fb9e4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:58.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.124+0000 7fb9e37fe640 1 -- 172.21.15.98:0/2627439210 >> v1:172.21.15.98:6789/0 conn(0x7fb9e414a700 legacy=0x7fb9e40abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:58.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9e37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9e37fe640 1 monclient: found mon.noname-b 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9e37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9e37fe640 1 -- 172.21.15.98:0/2627439210 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e40ac9d0 con 0x7fb9e40b0770 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9e37fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9e37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9eaf89640 5 monclient: authenticate success, global_id 4271 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9eaf89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9e37fe640 1 -- 172.21.15.98:0/2627439210 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb9dc001a10 con 0x7fb9e40b0770 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9e37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9e37fe640 10 monclient: dump: 2024-03-29T10:45:58.119 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.125+0000 7fb9eaf89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.127+0000 7fb9e37fe640 1 -- 172.21.15.98:0/2627439210 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9dc001ce0 con 0x7fb9e40b0770 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.127+0000 7fb9e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.127+0000 7fb9eaf89640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.127+0000 7fb9eaf89640 4 set_mon_vals no callback set 2024-03-29T10:45:58.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.127+0000 7fb9e37fe640 1 -- 172.21.15.98:0/2627439210 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb9dc002070 con 0x7fb9e40b0770 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.127+0000 7fb9e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.127+0000 7fb9e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.127+0000 7fb9eaf89640 10 monclient: shutdown 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.128+0000 7fb9e37fe640 10 monclient: dump: 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:58.121 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.128+0000 7fb9eaf89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.128+0000 7fb9eaf89640 1 -- 172.21.15.98:0/2627439210 >> v1:172.21.15.98:6790/0 conn(0x7fb9e40b0770 legacy=0x7fb9e4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.128+0000 7fb9eaf89640 1 -- 172.21.15.98:0/2627439210 shutdown_connections 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.128+0000 7fb9eaf89640 1 -- 172.21.15.98:0/2627439210 >> 172.21.15.98:0/2627439210 conn(0x7fb9e406b660 msgr2=0x7fb9e406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.128+0000 7fb9eaf89640 1 -- 172.21.15.98:0/2627439210 shutdown_connections 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.128+0000 7fb9eaf89640 1 -- 172.21.15.98:0/2627439210 wait complete. 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.129+0000 7fb9eaf89640 10 monclient: build_initial_monmap 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.129+0000 7fb9eaf89640 10 monclient: monmap: 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:58.122 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.129+0000 7fb9eaf89640 1 librados: starting msgr at 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.129+0000 7fb9eaf89640 1 librados: starting objecter 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.129+0000 7fb9eaf89640 1 Processor -- start 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.129+0000 7fb9eaf89640 1 -- start start 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.129+0000 7fb9eaf89640 1 librados: setting wanted keys 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.129+0000 7fb9eaf89640 1 librados: calling monclient init 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.129+0000 7fb9eaf89640 10 monclient: init 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.130+0000 7fb9eaf89640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.130+0000 7fb9eaf89640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.130+0000 7fb9eaf89640 10 monclient(hunting): picked mon.noname-c con 0x7fb9e40ac5a0 addr v1:172.21.15.98:6790/0 2024-03-29T10:45:58.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9e8d25640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fb9e40ac5a0 0x7fb9e41db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:35816/0 (socket says 172.21.15.98:35816) 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9eaf89640 10 monclient(hunting): picked mon.noname-a con 0x7fb9e40b0770 addr v1:172.21.15.98:6789/0 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9e8d25640 1 -- 172.21.15.98:0/3238953012 learned_addr learned my addr 172.21.15.98:0/3238953012 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9eaf89640 10 monclient(hunting): picked mon.noname-b con 0x7fb9e414a700 addr v1:172.21.15.189:6789/0 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9eaf89640 1 -- 172.21.15.98:0/3238953012 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e4125130 con 0x7fb9e40b0770 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9eaf89640 1 -- 172.21.15.98:0/3238953012 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e4147970 con 0x7fb9e414a700 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9e17fa640 10 client.?.objecter ms_handle_connect 0x7fb9e40ac5a0 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9eaf89640 1 -- 172.21.15.98:0/3238953012 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e4129060 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9eaf89640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9eaf89640 10 monclient(hunting): authenticate will time out at 978.328808s 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719918511 0 0) 0x7fb9e4129060 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9e17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9e17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9e17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:58.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9e17fa640 10 monclient(hunting): my global_id is 4274 2024-03-29T10:45:58.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.131+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d0004eb0 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.132+0000 7fb9e17fa640 10 client.?.objecter ms_handle_connect 0x7fb9e414a700 2024-03-29T10:45:58.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.132+0000 7fb9e17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:58.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.132+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2721427774 0 0) 0x7fb9d0004eb0 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.132+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9e4129060 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.132+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb9dc0019d0 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.132+0000 7fb9e17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:58.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.132+0000 7fb9e17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:58.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.132+0000 7fb9e17fa640 10 monclient(hunting): dump: 2024-03-29T10:45:58.125 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:58.125 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:58.125 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:58.125 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:58.126 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:58.126 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:58.126 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:58.126 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:58.126 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:58.126 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:58.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.132+0000 7fb9e17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:58.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.133+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4068059243 0 0) 0x7fb9e4129060 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.133+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 >> v1:172.21.15.189:6789/0 conn(0x7fb9e414a700 legacy=0x7fb9e41e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:58.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.133+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 >> v1:172.21.15.98:6789/0 conn(0x7fb9e40b0770 legacy=0x7fb9e4146cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:58.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.133+0000 7fb9e17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:45:58.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.133+0000 7fb9e17fa640 1 monclient: found mon.c 2024-03-29T10:45:58.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.133+0000 7fb9e17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.133+0000 7fb9e17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.133+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e41291f0 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.133+0000 7fb9e17fa640 10 monclient: _finish_auth 0 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.133+0000 7fb9e17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.133+0000 7fb9e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.134+0000 7fb9eaf89640 5 monclient: authenticate success, global_id 4274 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.134+0000 7fb9eaf89640 10 monclient: _renew_subs 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.134+0000 7fb9eaf89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.134+0000 7fb9eaf89640 1 -- 172.21.15.98:0/3238953012 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e41dc070 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.134+0000 7fb9eaf89640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.134+0000 7fb9eaf89640 10 monclient: _renew_subs 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.134+0000 7fb9eaf89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.134+0000 7fb9eaf89640 1 -- 172.21.15.98:0/3238953012 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e4147400 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.134+0000 7fb9eaf89640 1 librados: init done 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.135+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9dc004fb0 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.135+0000 7fb9e17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.135+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb9dc0052c0 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.135+0000 7fb9e17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.135+0000 7fb9e17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.135+0000 7fb9e17fa640 10 monclient: dump: 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:45:58.128 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:45:58.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.135+0000 7fb9eaf89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:58.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.136+0000 7fb9eaf89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:58.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.136+0000 7fb9eaf89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:58.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.136+0000 7fb9eaf89640 1 -- 172.21.15.98:0/3238953012 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e4075f10 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.136+0000 7fb9e2ffd640 4 set_mon_vals no callback set 2024-03-29T10:45:58.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.136+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fb9dc01b290 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.137+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fb9dc084eb0 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.137+0000 7fb9e17fa640 10 client.4274.objecter ms_dispatch 0x7fb9e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:58.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.137+0000 7fb9e17fa640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:58.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.137+0000 7fb9e17fa640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:58.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.137+0000 7fb9e17fa640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T10:45:58.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.138+0000 7fb9e3fff640 10 monclient: get_auth_request con 0x7fb9d006dee0 auth_method 0 2024-03-29T10:45:58.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.138+0000 7fb9e17fa640 10 client.4274.objecter ms_handle_connect 0x7fb9d006dee0 2024-03-29T10:45:58.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.143+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9dc050790 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.143+0000 7fb9e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:58.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.143+0000 7fb9e17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:58.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.447+0000 7fb9eaf89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:58.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.447+0000 7fb9eaf89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:58.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.447+0000 7fb9eaf89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:45:58.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.447+0000 7fb9eaf89640 1 -- 172.21.15.98:0/3238953012 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9e4147700 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.447+0000 7fb9e17fa640 1 -- 172.21.15.98:0/3238953012 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fb9dc005120 con 0x7fb9e40ac5a0 2024-03-29T10:45:58.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.447+0000 7fb9e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:58.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.447+0000 7fb9e17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:58.430 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:45:58.431 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:58.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.450+0000 7fb9eaf89640 10 librados: watch_flush enter 2024-03-29T10:45:58.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.451+0000 7fb9eaf89640 10 librados: watch_flush exit 2024-03-29T10:45:58.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.451+0000 7fb9eaf89640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T10:45:58.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.451+0000 7fb9eaf89640 10 client.4274.objecter successfully canceled tick 2024-03-29T10:45:58.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.451+0000 7fb9eaf89640 1 -- 172.21.15.98:0/3238953012 >> v1:172.21.15.98:6816/1481799426 conn(0x7fb9d006dee0 legacy=0x7fb9d00703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:58.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.451+0000 7fb9eaf89640 10 monclient: shutdown 2024-03-29T10:45:58.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.451+0000 7fb9eaf89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:58.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.451+0000 7fb9eaf89640 1 -- 172.21.15.98:0/3238953012 >> v1:172.21.15.98:6790/0 conn(0x7fb9e40ac5a0 legacy=0x7fb9e41db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:58.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.451+0000 7fb9eaf89640 1 -- 172.21.15.98:0/3238953012 shutdown_connections 2024-03-29T10:45:58.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.451+0000 7fb9eaf89640 1 -- 172.21.15.98:0/3238953012 >> 172.21.15.98:0/3238953012 conn(0x7fb9e406b660 msgr2=0x7fb9e41484c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:58.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.451+0000 7fb9eaf89640 1 -- 172.21.15.98:0/3238953012 shutdown_connections 2024-03-29T10:45:58.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.451+0000 7fb9eaf89640 1 -- 172.21.15.98:0/3238953012 wait complete. 2024-03-29T10:45:58.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:45:58.451+0000 7fb9eaf89640 1 librados: shutdown 2024-03-29T10:45:58.448 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:05.256 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:05.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.411+0000 7f9b43d54640 10 monclient: get_monmap_and_config 2024-03-29T10:46:05.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.411+0000 7f9b43d54640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.411+0000 7f9b43d54640 10 monclient: monmap: 2024-03-29T10:46:05.394 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:05.394 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:05.394 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:46:05.412468+0000 2024-03-29T10:46:05.394 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:46:05.412468+0000 2024-03-29T10:46:05.394 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.394 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:05.394 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:05.394 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:46:05.395 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:46:05.395 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:05.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.412+0000 7f9b43d54640 1 Processor -- start 2024-03-29T10:46:05.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.412+0000 7f9b43d54640 1 -- start start 2024-03-29T10:46:05.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.412+0000 7f9b43d54640 10 monclient: init 2024-03-29T10:46:05.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.413+0000 7f9b43d54640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.413+0000 7f9b43d54640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:05.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.413+0000 7f9b43d54640 10 monclient(hunting): picked mon.noname-a con 0x7f9b3c1422b0 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:05.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.413+0000 7f9b43d54640 10 monclient(hunting): picked mon.noname-c con 0x7f9b3c142690 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:05.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b422f1640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f9b3c0b17d0 0x7f9b3c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:45810/0 (socket says 172.21.15.98:45810) 2024-03-29T10:46:05.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b43d54640 10 monclient(hunting): picked mon.noname-b con 0x7f9b3c0b17d0 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:05.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b43d54640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c142c80 con 0x7f9b3c1422b0 2024-03-29T10:46:05.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b43d54640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c0aae40 con 0x7f9b3c142690 2024-03-29T10:46:05.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b43d54640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c13fbd0 con 0x7f9b3c0b17d0 2024-03-29T10:46:05.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b43d54640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b43d54640 10 monclient(hunting): authenticate will time out at 985.611113s 2024-03-29T10:46:05.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b422f1640 1 -- 172.21.15.98:0/212713762 learned_addr learned my addr 172.21.15.98:0/212713762 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:05.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b40aee640 1 -- 172.21.15.98:0/212713762 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718745627 0 0) 0x7f9b3c13fbd0 con 0x7f9b3c0b17d0 2024-03-29T10:46:05.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b40aee640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b40aee640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b40aee640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b40aee640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:46:05.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b40aee640 1 -- 172.21.15.98:0/212713762 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b28003520 con 0x7f9b3c0b17d0 2024-03-29T10:46:05.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b40aee640 1 -- 172.21.15.98:0/212713762 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2552586174 0 0) 0x7f9b28003520 con 0x7f9b3c0b17d0 2024-03-29T10:46:05.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b40aee640 1 -- 172.21.15.98:0/212713762 >> v1:172.21.15.189:6789/0 conn(0x7f9b3c142690 legacy=0x7f9b3c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:05.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.414+0000 7f9b40aee640 1 -- 172.21.15.98:0/212713762 >> v1:172.21.15.98:6789/0 conn(0x7f9b3c1422b0 legacy=0x7f9b3c13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:05.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 1 monclient: found mon.noname-b 2024-03-29T10:46:05.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:46:05.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 1 -- 172.21.15.98:0/212713762 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c151940 con 0x7f9b3c0b17d0 2024-03-29T10:46:05.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 1 -- 172.21.15.98:0/212713762 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f9b34001a10 con 0x7f9b3c0b17d0 2024-03-29T10:46:05.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b43d54640 5 monclient: authenticate success, global_id 4277 2024-03-29T10:46:05.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b43d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:05.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:05.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 10 monclient: dump: 2024-03-29T10:46:05.402 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:05.402 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:05.402 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:05.402 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:05.402 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.402 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:05.402 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b43d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 1 -- 172.21.15.98:0/212713762 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b34001b70 con 0x7f9b3c0b17d0 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 1 -- 172.21.15.98:0/212713762 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f9b34001fc0 con 0x7f9b3c0b17d0 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b43d54640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b43d54640 4 set_mon_vals no callback set 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.416+0000 7f9b40aee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.417+0000 7f9b40aee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.417+0000 7f9b40aee640 10 monclient: dump: 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:05.403 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:05.404 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:05.404 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:05.404 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.404 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:05.404 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:05.404 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:05.404 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:05.404 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:05.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.417+0000 7f9b43d54640 10 monclient: shutdown 2024-03-29T10:46:05.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.417+0000 7f9b43d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:05.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.417+0000 7f9b43d54640 1 -- 172.21.15.98:0/212713762 >> v1:172.21.15.98:6790/0 conn(0x7f9b3c0b17d0 legacy=0x7f9b3c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.418+0000 7f9b43d54640 1 -- 172.21.15.98:0/212713762 shutdown_connections 2024-03-29T10:46:05.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.418+0000 7f9b43d54640 1 -- 172.21.15.98:0/212713762 >> 172.21.15.98:0/212713762 conn(0x7f9b3c066a00 msgr2=0x7f9b3c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:05.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.418+0000 7f9b43d54640 1 -- 172.21.15.98:0/212713762 shutdown_connections 2024-03-29T10:46:05.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.418+0000 7f9b43d54640 1 -- 172.21.15.98:0/212713762 wait complete. 2024-03-29T10:46:05.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.419+0000 7f9b43d54640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.419+0000 7f9b43d54640 10 monclient: monmap: 2024-03-29T10:46:05.413 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:05.413 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:05.413 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:46:05.413 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:46:05.413 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.413 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:05.413 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:05.413 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:46:05.413 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:46:05.413 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:05.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.420+0000 7f9b43d54640 1 librados: starting msgr at 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.420+0000 7f9b43d54640 1 librados: starting objecter 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.420+0000 7f9b43d54640 1 Processor -- start 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.420+0000 7f9b43d54640 1 -- start start 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.420+0000 7f9b43d54640 1 librados: setting wanted keys 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.420+0000 7f9b43d54640 1 librados: calling monclient init 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.420+0000 7f9b43d54640 10 monclient: init 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.421+0000 7f9b43d54640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.421+0000 7f9b43d54640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.421+0000 7f9b43d54640 10 monclient(hunting): picked mon.noname-a con 0x7f9b3c0b17d0 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b43d54640 10 monclient(hunting): picked mon.noname-c con 0x7f9b3c1422b0 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b43d54640 10 monclient(hunting): picked mon.noname-b con 0x7f9b3c142690 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b43d54640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c1dfc50 con 0x7f9b3c0b17d0 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b43d54640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c06ecd0 con 0x7f9b3c142690 2024-03-29T10:46:05.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b43d54640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c1df020 con 0x7f9b3c1422b0 2024-03-29T10:46:05.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b43d54640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b43d54640 10 monclient(hunting): authenticate will time out at 985.619162s 2024-03-29T10:46:05.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b412ef640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f9b3c1422b0 0x7f9b3c070710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:46022/0 (socket says 172.21.15.98:46022) 2024-03-29T10:46:05.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b412ef640 1 -- 172.21.15.98:0/2395033537 learned_addr learned my addr 172.21.15.98:0/2395033537 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:05.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 10 client.?.objecter ms_handle_connect 0x7f9b3c1422b0 2024-03-29T10:46:05.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647271271 0 0) 0x7f9b3c1df020 con 0x7f9b3c1422b0 2024-03-29T10:46:05.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:46:05.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b20004eb0 con 0x7f9b3c1422b0 2024-03-29T10:46:05.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3476462265 0 0) 0x7f9b20004eb0 con 0x7f9b3c1422b0 2024-03-29T10:46:05.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b3c1df020 con 0x7f9b3c1422b0 2024-03-29T10:46:05.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f9b2c0017d0 con 0x7f9b3c1422b0 2024-03-29T10:46:05.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:05.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 10 monclient(hunting): dump: 2024-03-29T10:46:05.416 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:05.416 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:05.416 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:05.416 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.422+0000 7f9b327fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.423+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2896410400 0 0) 0x7f9b3c1df020 con 0x7f9b3c1422b0 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.423+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 >> v1:172.21.15.189:6789/0 conn(0x7f9b3c142690 legacy=0x7f9b3c1e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.423+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 >> v1:172.21.15.98:6789/0 conn(0x7f9b3c0b17d0 legacy=0x7f9b3c1df540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b327fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b327fc640 1 monclient: found mon.c 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:05.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c06e9f0 con 0x7f9b3c1422b0 2024-03-29T10:46:05.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b327fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b327fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b43d54640 5 monclient: authenticate success, global_id 4280 2024-03-29T10:46:05.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b43d54640 10 monclient: _renew_subs 2024-03-29T10:46:05.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b43d54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:05.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b43d54640 1 -- 172.21.15.98:0/2395033537 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3c0714b0 con 0x7f9b3c1422b0 2024-03-29T10:46:05.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b43d54640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:05.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b43d54640 10 monclient: _renew_subs 2024-03-29T10:46:05.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b43d54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:05.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b43d54640 1 -- 172.21.15.98:0/2395033537 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3c071850 con 0x7f9b3c1422b0 2024-03-29T10:46:05.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.424+0000 7f9b43d54640 1 librados: init done 2024-03-29T10:46:05.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.425+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b2c001930 con 0x7f9b3c1422b0 2024-03-29T10:46:05.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.425+0000 7f9b43d54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.425+0000 7f9b43d54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.425+0000 7f9b43d54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:05.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.425+0000 7f9b43d54640 1 -- 172.21.15.98:0/2395033537 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3c13bc20 con 0x7f9b3c1422b0 2024-03-29T10:46:05.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.426+0000 7f9b327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.426+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f9b2c004a40 con 0x7f9b3c1422b0 2024-03-29T10:46:05.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.426+0000 7f9b33fff640 4 set_mon_vals no callback set 2024-03-29T10:46:05.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.426+0000 7f9b327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.426+0000 7f9b327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.426+0000 7f9b327fc640 10 monclient: dump: 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.427+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f9b2c01aa10 con 0x7f9b3c1422b0 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.428+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f9b2c084a00 con 0x7f9b3c1422b0 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.428+0000 7f9b327fc640 10 client.4280.objecter ms_dispatch 0x7f9b3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:05.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.428+0000 7f9b327fc640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:05.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.428+0000 7f9b327fc640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:05.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.428+0000 7f9b327fc640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-29T10:46:05.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.428+0000 7f9b41af0640 10 monclient: get_auth_request con 0x7f9b2006dc80 auth_method 0 2024-03-29T10:46:05.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.429+0000 7f9b327fc640 10 client.4280.objecter ms_handle_connect 0x7f9b2006dc80 2024-03-29T10:46:05.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.433+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b2c050110 con 0x7f9b3c1422b0 2024-03-29T10:46:05.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.433+0000 7f9b327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.433+0000 7f9b327fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:05.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.734+0000 7f9b43d54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:05.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.734+0000 7f9b43d54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:05.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.734+0000 7f9b43d54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:05.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.734+0000 7f9b43d54640 1 -- 172.21.15.98:0/2395033537 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b3c1dee10 con 0x7f9b3c1422b0 2024-03-29T10:46:05.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.735+0000 7f9b327fc640 1 -- 172.21.15.98:0/2395033537 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f9b2c052210 con 0x7f9b3c1422b0 2024-03-29T10:46:05.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.735+0000 7f9b327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:05.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.735+0000 7f9b327fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:05.718 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:46:05.718 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:05.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.738+0000 7f9b43d54640 10 librados: watch_flush enter 2024-03-29T10:46:05.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.738+0000 7f9b43d54640 10 librados: watch_flush exit 2024-03-29T10:46:05.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.738+0000 7f9b43d54640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-29T10:46:05.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.738+0000 7f9b43d54640 10 client.4280.objecter successfully canceled tick 2024-03-29T10:46:05.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.739+0000 7f9b43d54640 1 -- 172.21.15.98:0/2395033537 >> v1:172.21.15.98:6816/1481799426 conn(0x7f9b2006dc80 legacy=0x7f9b20070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.739+0000 7f9b43d54640 10 monclient: shutdown 2024-03-29T10:46:05.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.739+0000 7f9b43d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:05.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.739+0000 7f9b43d54640 1 -- 172.21.15.98:0/2395033537 >> v1:172.21.15.98:6790/0 conn(0x7f9b3c1422b0 legacy=0x7f9b3c070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.739+0000 7f9b43d54640 1 -- 172.21.15.98:0/2395033537 shutdown_connections 2024-03-29T10:46:05.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.739+0000 7f9b43d54640 1 -- 172.21.15.98:0/2395033537 >> 172.21.15.98:0/2395033537 conn(0x7f9b3c066a00 msgr2=0x7f9b3c0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:05.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.739+0000 7f9b43d54640 1 -- 172.21.15.98:0/2395033537 shutdown_connections 2024-03-29T10:46:05.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.739+0000 7f9b43d54640 1 -- 172.21.15.98:0/2395033537 wait complete. 2024-03-29T10:46:05.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:05.739+0000 7f9b43d54640 1 librados: shutdown 2024-03-29T10:46:05.736 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:12.543 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:12.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.707+0000 7f40097b5640 10 monclient: get_monmap_and_config 2024-03-29T10:46:12.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.707+0000 7f40097b5640 10 monclient: build_initial_monmap 2024-03-29T10:46:12.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.707+0000 7f40097b5640 10 monclient: monmap: 2024-03-29T10:46:12.691 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:12.691 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:12.691 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:46:12.709064+0000 2024-03-29T10:46:12.691 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:46:12.709064+0000 2024-03-29T10:46:12.691 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:12.691 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:12.691 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:12.691 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:46:12.692 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:46:12.692 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:12.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.708+0000 7f40097b5640 1 Processor -- start 2024-03-29T10:46:12.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.708+0000 7f40097b5640 1 -- start start 2024-03-29T10:46:12.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.708+0000 7f40097b5640 10 monclient: init 2024-03-29T10:46:12.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.709+0000 7f40097b5640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:12.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.709+0000 7f40097b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:12.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.709+0000 7f40097b5640 10 monclient(hunting): picked mon.noname-b con 0x7f40041457f0 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:12.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.709+0000 7f40097b5640 10 monclient(hunting): picked mon.noname-a con 0x7f400413f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:12.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.709+0000 7f40097b5640 10 monclient(hunting): picked mon.noname-c con 0x7f40040a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:12.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.709+0000 7f40097b5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40040aaef0 con 0x7f400413f650 2024-03-29T10:46:12.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.709+0000 7f40097b5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4004145bd0 con 0x7f40040a9d70 2024-03-29T10:46:12.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.709+0000 7f40097b5640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4004145eb0 con 0x7f40041457f0 2024-03-29T10:46:12.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.709+0000 7f40097b5640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:12.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.709+0000 7f40097b5640 10 monclient(hunting): authenticate will time out at 992.907545s 2024-03-29T10:46:12.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.710+0000 7f4002ffd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f40041457f0 0x7f400413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:41354/0 (socket says 172.21.15.98:41354) 2024-03-29T10:46:12.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.710+0000 7f4002ffd640 1 -- 172.21.15.98:0/1964658114 learned_addr learned my addr 172.21.15.98:0/1964658114 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:12.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.711+0000 7f40027fc640 1 -- 172.21.15.98:0/1964658114 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031888172 0 0) 0x7f4004145eb0 con 0x7f40041457f0 2024-03-29T10:46:12.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.711+0000 7f40027fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:12.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.711+0000 7f40027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:12.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.711+0000 7f40027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:12.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.711+0000 7f40027fc640 10 monclient(hunting): my global_id is 4283 2024-03-29T10:46:12.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.711+0000 7f40027fc640 1 -- 172.21.15.98:0/1964658114 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ff0003520 con 0x7f40041457f0 2024-03-29T10:46:12.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.711+0000 7f40027fc640 1 -- 172.21.15.98:0/1964658114 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3362937448 0 0) 0x7f3ff0003520 con 0x7f40041457f0 2024-03-29T10:46:12.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.711+0000 7f40027fc640 1 -- 172.21.15.98:0/1964658114 >> v1:172.21.15.189:6789/0 conn(0x7f40040a9d70 legacy=0x7f400414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:12.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40027fc640 1 -- 172.21.15.98:0/1964658114 >> v1:172.21.15.98:6789/0 conn(0x7f400413f650 legacy=0x7f40040b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:12.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40027fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:12.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40027fc640 1 monclient: found mon.noname-b 2024-03-29T10:46:12.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:12.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40027fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:46:12.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40027fc640 1 -- 172.21.15.98:0/1964658114 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4004146340 con 0x7f40041457f0 2024-03-29T10:46:12.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40027fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:12.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40027fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:12.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:12.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40027fc640 1 -- 172.21.15.98:0/1964658114 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3fec001690 con 0x7f40041457f0 2024-03-29T10:46:12.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:12.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40027fc640 10 monclient: dump: 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40097b5640 5 monclient: authenticate success, global_id 4283 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.713+0000 7f40097b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:12.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.714+0000 7f40027fc640 1 -- 172.21.15.98:0/1964658114 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fec001920 con 0x7f40041457f0 2024-03-29T10:46:12.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.714+0000 7f40027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:12.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.714+0000 7f40027fc640 1 -- 172.21.15.98:0/1964658114 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3fec001cb0 con 0x7f40041457f0 2024-03-29T10:46:12.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.714+0000 7f40027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:12.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.714+0000 7f40027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:12.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.714+0000 7f40027fc640 10 monclient: dump: 2024-03-29T10:46:12.700 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:12.700 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:12.700 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:12.700 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:12.700 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:12.700 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:12.700 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:12.700 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:12.700 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:12.700 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:12.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.714+0000 7f40097b5640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:12.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.714+0000 7f40097b5640 4 set_mon_vals no callback set 2024-03-29T10:46:12.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.715+0000 7f40097b5640 10 monclient: shutdown 2024-03-29T10:46:12.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.715+0000 7f40097b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:12.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.715+0000 7f40097b5640 1 -- 172.21.15.98:0/1964658114 >> v1:172.21.15.98:6790/0 conn(0x7f40041457f0 legacy=0x7f400413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:12.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.715+0000 7f40097b5640 1 -- 172.21.15.98:0/1964658114 shutdown_connections 2024-03-29T10:46:12.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.715+0000 7f40097b5640 1 -- 172.21.15.98:0/1964658114 >> 172.21.15.98:0/1964658114 conn(0x7f4004066a00 msgr2=0x7f4004066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:12.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.717+0000 7f40097b5640 1 -- 172.21.15.98:0/1964658114 shutdown_connections 2024-03-29T10:46:12.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.717+0000 7f40097b5640 1 -- 172.21.15.98:0/1964658114 wait complete. 2024-03-29T10:46:12.704 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.719+0000 7f40097b5640 10 monclient: build_initial_monmap 2024-03-29T10:46:12.704 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.719+0000 7f40097b5640 10 monclient: monmap: 2024-03-29T10:46:12.704 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:12.704 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:12.705 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:46:12.705 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:46:12.705 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:12.705 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:12.705 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:12.705 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:46:12.705 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:46:12.705 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:12.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.719+0000 7f40097b5640 1 librados: starting msgr at 2024-03-29T10:46:12.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.719+0000 7f40097b5640 1 librados: starting objecter 2024-03-29T10:46:12.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.719+0000 7f40097b5640 1 Processor -- start 2024-03-29T10:46:12.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 1 -- start start 2024-03-29T10:46:12.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 1 librados: setting wanted keys 2024-03-29T10:46:12.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 1 librados: calling monclient init 2024-03-29T10:46:12.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 10 monclient: init 2024-03-29T10:46:12.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:12.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:12.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 10 monclient(hunting): picked mon.noname-b con 0x7f40040a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:12.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 10 monclient(hunting): picked mon.noname-a con 0x7f400413f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:12.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 10 monclient(hunting): picked mon.noname-c con 0x7f40041457f0 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:12.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4004135bb0 con 0x7f400413f650 2024-03-29T10:46:12.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4004141300 con 0x7f40040a9d70 2024-03-29T10:46:12.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40041415e0 con 0x7f40041457f0 2024-03-29T10:46:12.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:12.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.722+0000 7f40097b5640 10 monclient(hunting): authenticate will time out at 992.920887s 2024-03-29T10:46:12.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.724+0000 7f40037fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f40041457f0 0x7f4004130090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:41648/0 (socket says 172.21.15.98:41648) 2024-03-29T10:46:12.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.724+0000 7f40037fe640 1 -- 172.21.15.98:0/4283678005 learned_addr learned my addr 172.21.15.98:0/4283678005 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 10 client.?.objecter ms_handle_connect 0x7f40041457f0 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836277084 0 0) 0x7f40041415e0 con 0x7f40041457f0 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe4004eb0 con 0x7f40041457f0 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2349031556 0 0) 0x7f3fe4004eb0 con 0x7f40041457f0 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40041415e0 con 0x7f40041457f0 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3ffc001b70 con 0x7f40041457f0 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:12.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 10 monclient(hunting): dump: 2024-03-29T10:46:12.712 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:12.712 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:12.712 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:12.712 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:12.712 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:12.712 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:12.712 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:12.712 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:12.712 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:12.712 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:12.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.725+0000 7f3ffb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:12.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.726+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2303111897 0 0) 0x7f40041415e0 con 0x7f40041457f0 2024-03-29T10:46:12.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.726+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 >> v1:172.21.15.189:6789/0 conn(0x7f40040a9d70 legacy=0x7f4004141a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:12.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.726+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 >> v1:172.21.15.98:6789/0 conn(0x7f400413f650 legacy=0x7f400413fea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:12.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.727+0000 7f3ffb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:46:12.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.727+0000 7f3ffb7fe640 1 monclient: found mon.c 2024-03-29T10:46:12.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.727+0000 7f3ffb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:12.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.727+0000 7f3ffb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:12.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.727+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400413d930 con 0x7f40041457f0 2024-03-29T10:46:12.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.727+0000 7f3ffb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:46:12.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.727+0000 7f3ffb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:46:12.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.727+0000 7f3ffb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:12.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.727+0000 7f40097b5640 5 monclient: authenticate success, global_id 4286 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.727+0000 7f40097b5640 10 monclient: _renew_subs 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.727+0000 7f40097b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.727+0000 7f40097b5640 1 -- 172.21.15.98:0/4283678005 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4004135690 con 0x7f40041457f0 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.728+0000 7f40097b5640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.728+0000 7f40097b5640 10 monclient: _renew_subs 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.728+0000 7f40097b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.728+0000 7f40097b5640 1 -- 172.21.15.98:0/4283678005 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f400413e830 con 0x7f40041457f0 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.728+0000 7f40097b5640 1 librados: init done 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.729+0000 7f3ffb7fe640 10 client.4286.objecter ms_handle_connect 0x7f400413f650 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.729+0000 7f3ffb7fe640 10 client.4286.objecter resend_mon_ops 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.729+0000 7f40097b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.729+0000 7f40097b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.729+0000 7f40097b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.729+0000 7f40097b5640 1 -- 172.21.15.98:0/4283678005 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4004075f10 con 0x7f40041457f0 2024-03-29T10:46:12.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.729+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ffc001cd0 con 0x7f40041457f0 2024-03-29T10:46:12.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.729+0000 7f3ffb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:12.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.729+0000 7f40017fa640 4 set_mon_vals no callback set 2024-03-29T10:46:12.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.730+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3ffc004e20 con 0x7f40041457f0 2024-03-29T10:46:12.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.730+0000 7f3ffb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:12.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.730+0000 7f3ffb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:12.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.730+0000 7f3ffb7fe640 10 monclient: dump: 2024-03-29T10:46:12.719 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:12.719 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:12.719 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:12.719 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:12.720 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:12.720 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:12.720 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:12.720 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:12.720 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:12.720 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:12.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.732+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f3ffc01ae70 con 0x7f40041457f0 2024-03-29T10:46:12.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.733+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f3ffc084e20 con 0x7f40041457f0 2024-03-29T10:46:12.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.733+0000 7f3ffb7fe640 10 client.4286.objecter ms_dispatch 0x7f4004000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:12.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.733+0000 7f3ffb7fe640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:12.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.733+0000 7f3ffb7fe640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:12.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.733+0000 7f3ffb7fe640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-29T10:46:12.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.733+0000 7f4002ffd640 10 monclient: get_auth_request con 0x7f3fe406dc40 auth_method 0 2024-03-29T10:46:12.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.734+0000 7f3ffb7fe640 10 client.4286.objecter ms_handle_connect 0x7f3fe406dc40 2024-03-29T10:46:12.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.739+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ffc050530 con 0x7f40041457f0 2024-03-29T10:46:12.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.739+0000 7f3ffb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:12.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:12.739+0000 7f3ffb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:13.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.069+0000 7f40097b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:13.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.069+0000 7f40097b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:13.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.069+0000 7f40097b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:13.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.069+0000 7f40097b5640 1 -- 172.21.15.98:0/4283678005 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40040b3680 con 0x7f40041457f0 2024-03-29T10:46:13.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.069+0000 7f3ffb7fe640 1 -- 172.21.15.98:0/4283678005 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f3ffc052630 con 0x7f40041457f0 2024-03-29T10:46:13.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.069+0000 7f3ffb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:13.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.069+0000 7f3ffb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:13.053 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:46:13.054 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:13.056 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.073+0000 7f40097b5640 10 librados: watch_flush enter 2024-03-29T10:46:13.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.074+0000 7f40097b5640 10 librados: watch_flush exit 2024-03-29T10:46:13.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.074+0000 7f40097b5640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-29T10:46:13.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.074+0000 7f40097b5640 10 client.4286.objecter successfully canceled tick 2024-03-29T10:46:13.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.074+0000 7f40097b5640 1 -- 172.21.15.98:0/4283678005 >> v1:172.21.15.98:6816/1481799426 conn(0x7f3fe406dc40 legacy=0x7f3fe4070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:13.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.074+0000 7f40097b5640 10 monclient: shutdown 2024-03-29T10:46:13.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.074+0000 7f40097b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:13.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.074+0000 7f40097b5640 1 -- 172.21.15.98:0/4283678005 >> v1:172.21.15.98:6790/0 conn(0x7f40041457f0 legacy=0x7f4004130090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:13.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.074+0000 7f40097b5640 1 -- 172.21.15.98:0/4283678005 shutdown_connections 2024-03-29T10:46:13.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.074+0000 7f40097b5640 1 -- 172.21.15.98:0/4283678005 >> 172.21.15.98:0/4283678005 conn(0x7f4004066a00 msgr2=0x7f40040b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:13.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.074+0000 7f40097b5640 1 -- 172.21.15.98:0/4283678005 shutdown_connections 2024-03-29T10:46:13.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.074+0000 7f40097b5640 1 -- 172.21.15.98:0/4283678005 wait complete. 2024-03-29T10:46:13.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:13.076+0000 7f40097b5640 1 librados: shutdown 2024-03-29T10:46:13.076 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:19.884 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:20.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.048+0000 7fb237520640 10 monclient: get_monmap_and_config 2024-03-29T10:46:20.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.048+0000 7fb237520640 10 monclient: build_initial_monmap 2024-03-29T10:46:20.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.048+0000 7fb237520640 10 monclient: monmap: 2024-03-29T10:46:20.032 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:20.032 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:20.032 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:46:20.050007+0000 2024-03-29T10:46:20.033 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:46:20.050007+0000 2024-03-29T10:46:20.033 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:20.033 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:20.033 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:20.033 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:46:20.033 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:46:20.033 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:20.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.049+0000 7fb237520640 1 Processor -- start 2024-03-29T10:46:20.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.049+0000 7fb237520640 1 -- start start 2024-03-29T10:46:20.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.049+0000 7fb237520640 10 monclient: init 2024-03-29T10:46:20.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.050+0000 7fb237520640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:20.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.050+0000 7fb237520640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:20.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.050+0000 7fb237520640 10 monclient(hunting): picked mon.noname-b con 0x7fb2280389e0 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:20.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.050+0000 7fb237520640 10 monclient(hunting): picked mon.noname-a con 0x7fb228038dc0 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:20.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.050+0000 7fb23651e640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fb2280389e0 0x7fb228033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:33160/0 (socket says 172.21.15.98:33160) 2024-03-29T10:46:20.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.053+0000 7fb237520640 10 monclient(hunting): picked mon.noname-c con 0x7fb228030660 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:20.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.053+0000 7fb237520640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2280c54c0 con 0x7fb228038dc0 2024-03-29T10:46:20.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.053+0000 7fb237520640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb228036430 con 0x7fb228030660 2024-03-29T10:46:20.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.053+0000 7fb237520640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb228039840 con 0x7fb2280389e0 2024-03-29T10:46:20.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.053+0000 7fb23651e640 1 -- 172.21.15.98:0/328631895 learned_addr learned my addr 172.21.15.98:0/328631895 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:20.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.053+0000 7fb237520640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:20.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.053+0000 7fb237520640 10 monclient(hunting): authenticate will time out at 1000.251928s 2024-03-29T10:46:20.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.054+0000 7fb23551c640 1 -- 172.21.15.98:0/328631895 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683995012 0 0) 0x7fb228039840 con 0x7fb2280389e0 2024-03-29T10:46:20.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.054+0000 7fb23551c640 10 monclient(hunting): _negotiate 2024-03-29T10:46:20.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.054+0000 7fb23551c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:20.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.054+0000 7fb23551c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:20.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.054+0000 7fb23551c640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:46:20.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.054+0000 7fb23551c640 1 -- 172.21.15.98:0/328631895 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb21c003520 con 0x7fb2280389e0 2024-03-29T10:46:20.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.054+0000 7fb23551c640 1 -- 172.21.15.98:0/328631895 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344295276 0 0) 0x7fb228036430 con 0x7fb228030660 2024-03-29T10:46:20.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.054+0000 7fb23551c640 10 monclient(hunting): _negotiate 2024-03-29T10:46:20.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.054+0000 7fb23551c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:20.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.054+0000 7fb23551c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:20.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.054+0000 7fb23551c640 10 monclient(hunting): my global_id is 4285 2024-03-29T10:46:20.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.054+0000 7fb23551c640 1 -- 172.21.15.98:0/328631895 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb228039840 con 0x7fb228030660 2024-03-29T10:46:20.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 1 -- 172.21.15.98:0/328631895 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3147941720 0 0) 0x7fb21c003520 con 0x7fb2280389e0 2024-03-29T10:46:20.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 1 -- 172.21.15.98:0/328631895 >> v1:172.21.15.189:6789/0 conn(0x7fb228030660 legacy=0x7fb2280d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:20.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 1 -- 172.21.15.98:0/328631895 >> v1:172.21.15.98:6789/0 conn(0x7fb228038dc0 legacy=0x7fb2280cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:20.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 10 monclient: _finish_hunting 0 2024-03-29T10:46:20.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 1 monclient: found mon.noname-b 2024-03-29T10:46:20.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:20.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:46:20.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 1 -- 172.21.15.98:0/328631895 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb228035690 con 0x7fb2280389e0 2024-03-29T10:46:20.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 10 monclient: _finish_auth 0 2024-03-29T10:46:20.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 10 monclient: _check_auth_tickets 2024-03-29T10:46:20.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:20.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 1 -- 172.21.15.98:0/328631895 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb22c001690 con 0x7fb2280389e0 2024-03-29T10:46:20.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:20.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:20.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb23551c640 10 monclient: dump: 2024-03-29T10:46:20.044 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:20.044 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:20.044 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:20.044 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:20.044 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:20.044 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:20.044 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:20.044 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:20.044 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:20.044 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:20.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb237520640 5 monclient: authenticate success, global_id 4289 2024-03-29T10:46:20.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.055+0000 7fb237520640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:20.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.057+0000 7fb23551c640 1 -- 172.21.15.98:0/328631895 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb22c001860 con 0x7fb2280389e0 2024-03-29T10:46:20.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.057+0000 7fb23551c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:20.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.057+0000 7fb23551c640 1 -- 172.21.15.98:0/328631895 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb22c001cb0 con 0x7fb2280389e0 2024-03-29T10:46:20.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.057+0000 7fb23551c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:20.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.057+0000 7fb23551c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:20.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.057+0000 7fb23551c640 10 monclient: dump: 2024-03-29T10:46:20.045 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:20.045 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:20.045 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:20.045 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:20.045 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:20.046 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:20.046 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:20.046 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:20.046 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:20.046 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:20.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.058+0000 7fb237520640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:20.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.058+0000 7fb237520640 4 set_mon_vals no callback set 2024-03-29T10:46:20.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.058+0000 7fb237520640 10 monclient: shutdown 2024-03-29T10:46:20.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.059+0000 7fb237520640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:20.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.059+0000 7fb237520640 1 -- 172.21.15.98:0/328631895 >> v1:172.21.15.98:6790/0 conn(0x7fb2280389e0 legacy=0x7fb228033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:20.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.062+0000 7fb237520640 1 -- 172.21.15.98:0/328631895 shutdown_connections 2024-03-29T10:46:20.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.062+0000 7fb237520640 1 -- 172.21.15.98:0/328631895 >> 172.21.15.98:0/328631895 conn(0x7fb228005c00 msgr2=0x7fb228006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:20.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.062+0000 7fb237520640 1 -- 172.21.15.98:0/328631895 shutdown_connections 2024-03-29T10:46:20.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.065+0000 7fb237520640 1 -- 172.21.15.98:0/328631895 wait complete. 2024-03-29T10:46:20.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.066+0000 7fb237520640 10 monclient: build_initial_monmap 2024-03-29T10:46:20.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.066+0000 7fb237520640 10 monclient: monmap: 2024-03-29T10:46:20.051 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:20.051 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:20.051 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:46:20.051 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:46:20.051 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:20.051 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:20.051 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:20.052 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:46:20.052 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:46:20.052 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:20.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.067+0000 7fb237520640 1 librados: starting msgr at 2024-03-29T10:46:20.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.067+0000 7fb237520640 1 librados: starting objecter 2024-03-29T10:46:20.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.067+0000 7fb237520640 1 Processor -- start 2024-03-29T10:46:20.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.067+0000 7fb237520640 1 -- start start 2024-03-29T10:46:20.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.067+0000 7fb237520640 1 librados: setting wanted keys 2024-03-29T10:46:20.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.067+0000 7fb237520640 1 librados: calling monclient init 2024-03-29T10:46:20.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.067+0000 7fb237520640 10 monclient: init 2024-03-29T10:46:20.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.068+0000 7fb237520640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:20.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.068+0000 7fb237520640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:20.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.069+0000 7fb237520640 10 monclient(hunting): picked mon.noname-c con 0x7fb228030660 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:20.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.070+0000 7fb237520640 10 monclient(hunting): picked mon.noname-b con 0x7fb2280389e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:20.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.070+0000 7fb237520640 10 monclient(hunting): picked mon.noname-a con 0x7fb2280c5b10 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:20.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.070+0000 7fb237520640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2280c7800 con 0x7fb2280c5b10 2024-03-29T10:46:20.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.070+0000 7fb237520640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2280c73e0 con 0x7fb2280389e0 2024-03-29T10:46:20.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.070+0000 7fb237520640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22815c8e0 con 0x7fb228030660 2024-03-29T10:46:20.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.070+0000 7fb237520640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:20.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.070+0000 7fb237520640 10 monclient(hunting): authenticate will time out at 1000.268540s 2024-03-29T10:46:20.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.070+0000 7fb23651e640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fb228030660 0x7fb22802fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:36282/0 (socket says 172.21.15.98:36282) 2024-03-29T10:46:20.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.070+0000 7fb23651e640 1 -- 172.21.15.98:0/845740674 learned_addr learned my addr 172.21.15.98:0/845740674 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:20.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.071+0000 7fb226ffd640 10 client.?.objecter ms_handle_connect 0x7fb228030660 2024-03-29T10:46:20.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.071+0000 7fb226ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:20.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.072+0000 7fb226ffd640 10 client.?.objecter ms_handle_connect 0x7fb2280389e0 2024-03-29T10:46:20.055 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.072+0000 7fb226ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:20.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.073+0000 7fb226ffd640 10 client.?.objecter ms_handle_connect 0x7fb2280c5b10 2024-03-29T10:46:20.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.073+0000 7fb226ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:20.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413273818 0 0) 0x7fb22815c8e0 con 0x7fb228030660 2024-03-29T10:46:20.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:20.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:20.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:20.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:46:20.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb214004eb0 con 0x7fb228030660 2024-03-29T10:46:20.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741747768 0 0) 0x7fb2280c73e0 con 0x7fb2280389e0 2024-03-29T10:46:20.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:20.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:20.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:20.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 10 monclient(hunting): my global_id is 4288 2024-03-29T10:46:20.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb22815c8e0 con 0x7fb2280389e0 2024-03-29T10:46:20.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900050752 0 0) 0x7fb2280c7800 con 0x7fb2280c5b10 2024-03-29T10:46:20.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:20.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:20.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:20.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.075+0000 7fb226ffd640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:46:20.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.076+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2280c73e0 con 0x7fb2280c5b10 2024-03-29T10:46:20.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.076+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3726531647 0 0) 0x7fb214004eb0 con 0x7fb228030660 2024-03-29T10:46:20.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.076+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2280c7800 con 0x7fb228030660 2024-03-29T10:46:20.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.076+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 997428446 0 0) 0x7fb22815c8e0 con 0x7fb2280389e0 2024-03-29T10:46:20.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.076+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb214004eb0 con 0x7fb2280389e0 2024-03-29T10:46:20.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.076+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb22c001b70 con 0x7fb228030660 2024-03-29T10:46:20.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.076+0000 7fb226ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:20.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.076+0000 7fb226ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:20.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.076+0000 7fb226ffd640 10 monclient(hunting): dump: 2024-03-29T10:46:20.069 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:20.069 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:20.069 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:20.069 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:20.070 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:20.070 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:20.070 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:20.070 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:20.070 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:20.070 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:20.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.076+0000 7fb226ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:20.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb220001710 con 0x7fb2280389e0 2024-03-29T10:46:20.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:20.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:20.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 10 monclient(hunting): dump: 2024-03-29T10:46:20.071 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:20.071 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:20.072 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:20.072 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:20.072 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:20.072 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:20.072 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:20.072 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:20.073 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:20.073 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:20.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1465384161 0 0) 0x7fb2280c73e0 con 0x7fb2280c5b10 2024-03-29T10:46:20.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb22815c8e0 con 0x7fb2280c5b10 2024-03-29T10:46:20.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb23004fdd0 con 0x7fb2280c5b10 2024-03-29T10:46:20.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:20.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:20.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 10 monclient(hunting): dump: 2024-03-29T10:46:20.074 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:20.074 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:20.074 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:20.074 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:20.074 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:20.074 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:20.074 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:20.075 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:20.075 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:20.075 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:20.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1339636745 0 0) 0x7fb2280c7800 con 0x7fb228030660 2024-03-29T10:46:20.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 >> v1:172.21.15.189:6789/0 conn(0x7fb2280389e0 legacy=0x7fb2280c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:20.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 >> v1:172.21.15.98:6789/0 conn(0x7fb2280c5b10 legacy=0x7fb22815cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:20.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:20.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 1 monclient: found mon.c 2024-03-29T10:46:20.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.077+0000 7fb226ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:20.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.078+0000 7fb226ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:20.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.078+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2280c61d0 con 0x7fb228030660 2024-03-29T10:46:20.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.078+0000 7fb226ffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:20.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.078+0000 7fb226ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:20.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.078+0000 7fb226ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:20.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.078+0000 7fb237520640 5 monclient: authenticate success, global_id 4292 2024-03-29T10:46:20.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.078+0000 7fb237520640 10 monclient: _renew_subs 2024-03-29T10:46:20.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.078+0000 7fb237520640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:20.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.078+0000 7fb237520640 1 -- 172.21.15.98:0/845740674 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2280c83f0 con 0x7fb228030660 2024-03-29T10:46:20.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.078+0000 7fb237520640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:20.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.078+0000 7fb237520640 10 monclient: _renew_subs 2024-03-29T10:46:20.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.079+0000 7fb237520640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:20.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.079+0000 7fb237520640 1 -- 172.21.15.98:0/845740674 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22815d4e0 con 0x7fb228030660 2024-03-29T10:46:20.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.079+0000 7fb237520640 1 librados: init done 2024-03-29T10:46:20.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.079+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb22c004bf0 con 0x7fb228030660 2024-03-29T10:46:20.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.079+0000 7fb226ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:20.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.080+0000 7fb234d1b640 4 set_mon_vals no callback set 2024-03-29T10:46:20.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.080+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb22c004f00 con 0x7fb228030660 2024-03-29T10:46:20.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.080+0000 7fb226ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:20.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.080+0000 7fb237520640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:20.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.080+0000 7fb226ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:20.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.080+0000 7fb226ffd640 10 monclient: dump: 2024-03-29T10:46:20.080 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:20.080 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:20.080 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:20.080 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:20.080 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:20.080 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:20.080 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:20.081 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:20.081 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:20.081 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:20.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.081+0000 7fb237520640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:20.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.081+0000 7fb237520640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:20.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.081+0000 7fb237520640 1 -- 172.21.15.98:0/845740674 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb218005080 con 0x7fb228030660 2024-03-29T10:46:20.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.081+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fb22c01aed0 con 0x7fb228030660 2024-03-29T10:46:20.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.082+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fb22c084b20 con 0x7fb228030660 2024-03-29T10:46:20.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.082+0000 7fb226ffd640 10 client.4292.objecter ms_dispatch 0x7fb230000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:20.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.082+0000 7fb226ffd640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:20.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.082+0000 7fb226ffd640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:20.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.082+0000 7fb226ffd640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T10:46:20.084 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.083+0000 7fb235d1d640 10 monclient: get_auth_request con 0x7fb21406dea0 auth_method 0 2024-03-29T10:46:20.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.083+0000 7fb226ffd640 10 client.4292.objecter ms_handle_connect 0x7fb21406dea0 2024-03-29T10:46:20.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.090+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb22c050400 con 0x7fb228030660 2024-03-29T10:46:20.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.090+0000 7fb226ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:20.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.090+0000 7fb226ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:20.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.419+0000 7fb237520640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:20.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.419+0000 7fb237520640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:20.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.419+0000 7fb237520640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:46:20.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.419+0000 7fb237520640 1 -- 172.21.15.98:0/845740674 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb218005330 con 0x7fb228030660 2024-03-29T10:46:20.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.421+0000 7fb226ffd640 1 -- 172.21.15.98:0/845740674 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fb22c052330 con 0x7fb228030660 2024-03-29T10:46:20.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.421+0000 7fb226ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:20.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.421+0000 7fb226ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:20.406 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:46:20.406 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:20.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.426+0000 7fb237520640 10 librados: watch_flush enter 2024-03-29T10:46:20.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.427+0000 7fb237520640 10 librados: watch_flush exit 2024-03-29T10:46:20.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.427+0000 7fb237520640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T10:46:20.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.427+0000 7fb237520640 10 client.4292.objecter successfully canceled tick 2024-03-29T10:46:20.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.427+0000 7fb237520640 1 -- 172.21.15.98:0/845740674 >> v1:172.21.15.98:6816/1481799426 conn(0x7fb21406dea0 legacy=0x7fb214070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:20.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.427+0000 7fb237520640 10 monclient: shutdown 2024-03-29T10:46:20.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.427+0000 7fb237520640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:20.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.427+0000 7fb237520640 1 -- 172.21.15.98:0/845740674 >> v1:172.21.15.98:6790/0 conn(0x7fb228030660 legacy=0x7fb22802fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:20.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.427+0000 7fb237520640 1 -- 172.21.15.98:0/845740674 shutdown_connections 2024-03-29T10:46:20.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.427+0000 7fb237520640 1 -- 172.21.15.98:0/845740674 >> 172.21.15.98:0/845740674 conn(0x7fb228005c00 msgr2=0x7fb228034b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:20.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.427+0000 7fb237520640 1 -- 172.21.15.98:0/845740674 shutdown_connections 2024-03-29T10:46:20.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.428+0000 7fb237520640 1 -- 172.21.15.98:0/845740674 wait complete. 2024-03-29T10:46:20.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:20.428+0000 7fb237520640 1 librados: shutdown 2024-03-29T10:46:20.427 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:27.235 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:27.383 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.400+0000 7f2f1620f640 10 monclient: get_monmap_and_config 2024-03-29T10:46:27.383 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.400+0000 7f2f1620f640 10 monclient: build_initial_monmap 2024-03-29T10:46:27.383 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.400+0000 7f2f1620f640 10 monclient: monmap: 2024-03-29T10:46:27.383 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:27.383 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:27.383 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:46:27.401317+0000 2024-03-29T10:46:27.383 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:46:27.401317+0000 2024-03-29T10:46:27.383 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:27.383 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:27.384 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:27.384 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:46:27.384 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:46:27.384 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:27.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.402+0000 7f2f1620f640 1 Processor -- start 2024-03-29T10:46:27.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.402+0000 7f2f1620f640 1 -- start start 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.402+0000 7f2f1620f640 10 monclient: init 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.403+0000 7f2f1620f640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.403+0000 7f2f1620f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.403+0000 7f2f1620f640 10 monclient(hunting): picked mon.noname-a con 0x7f2f100b0860 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.403+0000 7f2f1620f640 10 monclient(hunting): picked mon.noname-b con 0x7f2f1013ef30 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.403+0000 7f2f1620f640 10 monclient(hunting): picked mon.noname-c con 0x7f2f100af8e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.403+0000 7f2f1620f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f100b0c40 con 0x7f2f100b0860 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.403+0000 7f2f1620f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f100aed10 con 0x7f2f100af8e0 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.403+0000 7f2f1620f640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f100afd50 con 0x7f2f1013ef30 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.403+0000 7f2f1620f640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.403+0000 7f2f1620f640 10 monclient(hunting): authenticate will time out at 1007.600281s 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.403+0000 7f2f0effd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f2f1013ef30 0x7f2f100af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:53130/0 (socket says 172.21.15.98:53130) 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.403+0000 7f2f0effd640 1 -- 172.21.15.98:0/1434706173 learned_addr learned my addr 172.21.15.98:0/1434706173 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:27.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.403+0000 7f2f0e7fc640 1 -- 172.21.15.98:0/1434706173 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125228264 0 0) 0x7f2f100afd50 con 0x7f2f1013ef30 2024-03-29T10:46:27.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.404+0000 7f2f0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:27.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.404+0000 7f2f0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:27.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.404+0000 7f2f0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:27.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.404+0000 7f2f0e7fc640 10 monclient(hunting): my global_id is 4295 2024-03-29T10:46:27.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.404+0000 7f2f0e7fc640 1 -- 172.21.15.98:0/1434706173 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef8003520 con 0x7f2f1013ef30 2024-03-29T10:46:27.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.404+0000 7f2f0e7fc640 1 -- 172.21.15.98:0/1434706173 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2387831510 0 0) 0x7f2ef8003520 con 0x7f2f1013ef30 2024-03-29T10:46:27.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.404+0000 7f2f0e7fc640 1 -- 172.21.15.98:0/1434706173 >> v1:172.21.15.189:6789/0 conn(0x7f2f100af8e0 legacy=0x7f2f10149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:27.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.405+0000 7f2f0e7fc640 1 -- 172.21.15.98:0/1434706173 >> v1:172.21.15.98:6789/0 conn(0x7f2f100b0860 legacy=0x7f2f1013e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:27.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:27.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 1 monclient: found mon.noname-b 2024-03-29T10:46:27.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:27.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:46:27.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 1 -- 172.21.15.98:0/1434706173 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f100ab1b0 con 0x7f2f1013ef30 2024-03-29T10:46:27.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:27.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:27.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:27.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 1 -- 172.21.15.98:0/1434706173 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2f04001690 con 0x7f2f1013ef30 2024-03-29T10:46:27.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:27.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:27.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 10 monclient: dump: 2024-03-29T10:46:27.394 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:27.394 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:27.394 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 1 -- 172.21.15.98:0/1434706173 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f040018f0 con 0x7f2f1013ef30 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f1620f640 5 monclient: authenticate success, global_id 4295 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f1620f640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f1620f640 4 set_mon_vals no callback set 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 1 -- 172.21.15.98:0/1434706173 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2f04001d40 con 0x7f2f1013ef30 2024-03-29T10:46:27.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.408+0000 7f2f0e7fc640 10 monclient: dump: 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.410+0000 7f2f1620f640 10 monclient: shutdown 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.410+0000 7f2f1620f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:27.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.410+0000 7f2f1620f640 1 -- 172.21.15.98:0/1434706173 >> v1:172.21.15.98:6790/0 conn(0x7f2f1013ef30 legacy=0x7f2f100af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.410+0000 7f2f1620f640 1 -- 172.21.15.98:0/1434706173 shutdown_connections 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.410+0000 7f2f1620f640 1 -- 172.21.15.98:0/1434706173 >> 172.21.15.98:0/1434706173 conn(0x7f2f1006b660 msgr2=0x7f2f1006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.410+0000 7f2f1620f640 1 -- 172.21.15.98:0/1434706173 shutdown_connections 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.410+0000 7f2f1620f640 1 -- 172.21.15.98:0/1434706173 wait complete. 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.411+0000 7f2f1620f640 10 monclient: build_initial_monmap 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.411+0000 7f2f1620f640 10 monclient: monmap: 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:46:27.397 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:46:27.398 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:27.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.412+0000 7f2f1620f640 1 librados: starting msgr at 2024-03-29T10:46:27.404 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.412+0000 7f2f1620f640 1 librados: starting objecter 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.412+0000 7f2f1620f640 1 Processor -- start 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.412+0000 7f2f1620f640 1 -- start start 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.412+0000 7f2f1620f640 1 librados: setting wanted keys 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.412+0000 7f2f1620f640 1 librados: calling monclient init 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.412+0000 7f2f1620f640 10 monclient: init 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.413+0000 7f2f1620f640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.413+0000 7f2f1620f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.413+0000 7f2f1620f640 10 monclient(hunting): picked mon.noname-b con 0x7f2f100af8e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.413+0000 7f2f1620f640 10 monclient(hunting): picked mon.noname-c con 0x7f2f100b0860 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.413+0000 7f2f1620f640 10 monclient(hunting): picked mon.noname-a con 0x7f2f1013ef30 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.413+0000 7f2f1620f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1006fe20 con 0x7f2f1013ef30 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.413+0000 7f2f1620f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f100b2650 con 0x7f2f100af8e0 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.414+0000 7f2f1620f640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f10071c10 con 0x7f2f100b0860 2024-03-29T10:46:27.405 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.414+0000 7f2f1620f640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:27.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.414+0000 7f2f1620f640 10 monclient(hunting): authenticate will time out at 1007.610984s 2024-03-29T10:46:27.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.417+0000 7f2f0effd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f2f100b0860 0x7f2f101db8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:47374/0 (socket says 172.21.15.98:47374) 2024-03-29T10:46:27.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.417+0000 7f2f0effd640 1 -- 172.21.15.98:0/916881289 learned_addr learned my addr 172.21.15.98:0/916881289 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:27.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.417+0000 7f2ef7fff640 10 client.?.objecter ms_handle_connect 0x7f2f100b0860 2024-03-29T10:46:27.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:27.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 10 client.?.objecter ms_handle_connect 0x7f2f100af8e0 2024-03-29T10:46:27.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:27.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512060759 0 0) 0x7f2f100b2650 con 0x7f2f100af8e0 2024-03-29T10:46:27.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:27.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:27.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:27.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:46:27.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2efc004eb0 con 0x7f2f100af8e0 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183500721 0 0) 0x7f2f10071c10 con 0x7f2f100b0860 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.418+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f100b2650 con 0x7f2f100b0860 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.419+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2885205819 0 0) 0x7f2f100b2650 con 0x7f2f100b0860 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.419+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f10071c10 con 0x7f2f100b0860 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.419+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2f04001d40 con 0x7f2f100b0860 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.419+0000 7f2ef7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.419+0000 7f2ef7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.419+0000 7f2ef7fff640 10 monclient(hunting): dump: 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:27.407 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.419+0000 7f2ef7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.419+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 946440040 0 0) 0x7f2efc004eb0 con 0x7f2f100af8e0 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.419+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f100b2650 con 0x7f2f100af8e0 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2f00001510 con 0x7f2f100af8e0 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 10 monclient(hunting): dump: 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:27.408 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 10 client.?.objecter ms_handle_connect 0x7f2f1013ef30 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038252511 0 0) 0x7f2f1006fe20 con 0x7f2f1013ef30 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:27.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2efc004eb0 con 0x7f2f1013ef30 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4142398830 0 0) 0x7f2f100b2650 con 0x7f2f100af8e0 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.420+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 >> v1:172.21.15.98:6790/0 conn(0x7f2f100b0860 legacy=0x7f2f101db8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.421+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 >> v1:172.21.15.98:6789/0 conn(0x7f2f1013ef30 legacy=0x7f2f101deff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.421+0000 7f2ef7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.421+0000 7f2ef7fff640 1 monclient: found mon.b 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.421+0000 7f2ef7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.421+0000 7f2ef7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.421+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f10070090 con 0x7f2f100af8e0 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.421+0000 7f2ef7fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.421+0000 7f2ef7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.421+0000 7f2ef7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.421+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f00001670 con 0x7f2f100af8e0 2024-03-29T10:46:27.410 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.421+0000 7f2ef7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.421+0000 7f2f0dffb640 4 set_mon_vals no callback set 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.422+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2f00001d80 con 0x7f2f100af8e0 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.422+0000 7f2ef7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.422+0000 7f2ef7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.422+0000 7f2ef7fff640 10 monclient: dump: 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:27.411 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:27.412 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:27.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.422+0000 7f2f1620f640 5 monclient: authenticate success, global_id 4291 2024-03-29T10:46:27.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.422+0000 7f2f1620f640 10 monclient: _renew_subs 2024-03-29T10:46:27.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.422+0000 7f2f1620f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:27.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.422+0000 7f2f1620f640 1 -- 172.21.15.98:0/916881289 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f10070370 con 0x7f2f100af8e0 2024-03-29T10:46:27.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.422+0000 7f2f1620f640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:27.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.422+0000 7f2f1620f640 10 monclient: _renew_subs 2024-03-29T10:46:27.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.422+0000 7f2f1620f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:27.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.422+0000 7f2f1620f640 1 -- 172.21.15.98:0/916881289 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f100717a0 con 0x7f2f100af8e0 2024-03-29T10:46:27.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.422+0000 7f2f1620f640 1 librados: init done 2024-03-29T10:46:27.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.423+0000 7f2f1620f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:27.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.423+0000 7f2f1620f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:27.413 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.423+0000 7f2f1620f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:27.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.423+0000 7f2f1620f640 1 -- 172.21.15.98:0/916881289 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f10075f10 con 0x7f2f100af8e0 2024-03-29T10:46:27.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.427+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f2f00019780 con 0x7f2f100af8e0 2024-03-29T10:46:27.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.428+0000 7f2f0effd640 10 monclient: get_auth_request con 0x7f2efc06df90 auth_method 0 2024-03-29T10:46:27.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.428+0000 7f2ef7fff640 10 client.4291.objecter ms_handle_connect 0x7f2efc06df90 2024-03-29T10:46:27.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.429+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f2f00083760 con 0x7f2f100af8e0 2024-03-29T10:46:27.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.429+0000 7f2ef7fff640 10 client.4291.objecter ms_dispatch 0x7f2f10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:27.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.429+0000 7f2ef7fff640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:27.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.429+0000 7f2ef7fff640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:27.414 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.429+0000 7f2ef7fff640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-29T10:46:27.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.435+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f0004ee70 con 0x7f2f100af8e0 2024-03-29T10:46:27.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.435+0000 7f2ef7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:27.418 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.435+0000 7f2ef7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:27.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.765+0000 7f2f1620f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:27.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.765+0000 7f2f1620f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:27.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.765+0000 7f2f1620f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:27.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.765+0000 7f2f1620f640 1 -- 172.21.15.98:0/916881289 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f10072330 con 0x7f2f100af8e0 2024-03-29T10:46:27.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.770+0000 7f2ef7fff640 1 -- 172.21.15.98:0/916881289 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f2f00050f70 con 0x7f2f100af8e0 2024-03-29T10:46:27.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.770+0000 7f2ef7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:27.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.770+0000 7f2ef7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:27.755 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:46:27.755 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:27.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.775+0000 7f2f1620f640 10 librados: watch_flush enter 2024-03-29T10:46:27.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.776+0000 7f2f1620f640 10 librados: watch_flush exit 2024-03-29T10:46:27.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.776+0000 7f2f1620f640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-29T10:46:27.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.776+0000 7f2f1620f640 10 client.4291.objecter successfully canceled tick 2024-03-29T10:46:27.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.776+0000 7f2f1620f640 1 -- 172.21.15.98:0/916881289 >> v1:172.21.15.98:6816/1481799426 conn(0x7f2efc06df90 legacy=0x7f2efc070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:27.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.777+0000 7f2f1620f640 10 monclient: shutdown 2024-03-29T10:46:27.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.777+0000 7f2f1620f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:27.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.777+0000 7f2f1620f640 1 -- 172.21.15.98:0/916881289 >> v1:172.21.15.189:6789/0 conn(0x7f2f100af8e0 legacy=0x7f2f100b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:27.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.777+0000 7f2f1620f640 1 -- 172.21.15.98:0/916881289 shutdown_connections 2024-03-29T10:46:27.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.777+0000 7f2f1620f640 1 -- 172.21.15.98:0/916881289 >> 172.21.15.98:0/916881289 conn(0x7f2f1006b660 msgr2=0x7f2f1013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:27.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.777+0000 7f2f1620f640 1 -- 172.21.15.98:0/916881289 shutdown_connections 2024-03-29T10:46:27.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.777+0000 7f2f1620f640 1 -- 172.21.15.98:0/916881289 wait complete. 2024-03-29T10:46:27.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:27.777+0000 7f2f1620f640 1 librados: shutdown 2024-03-29T10:46:27.780 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:34.586 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:34.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.755+0000 7f1bb75ac640 10 monclient: get_monmap_and_config 2024-03-29T10:46:34.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.755+0000 7f1bb75ac640 10 monclient: build_initial_monmap 2024-03-29T10:46:34.739 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.755+0000 7f1bb75ac640 10 monclient: monmap: 2024-03-29T10:46:34.739 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:34.739 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:34.740 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:46:34.756488+0000 2024-03-29T10:46:34.740 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:46:34.756488+0000 2024-03-29T10:46:34.740 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:34.740 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:34.740 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:34.740 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:46:34.740 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:46:34.740 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:34.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.756+0000 7f1bb75ac640 1 Processor -- start 2024-03-29T10:46:34.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.756+0000 7f1bb75ac640 1 -- start start 2024-03-29T10:46:34.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.756+0000 7f1bb75ac640 10 monclient: init 2024-03-29T10:46:34.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.757+0000 7f1bb75ac640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:34.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.757+0000 7f1bb75ac640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:34.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.757+0000 7f1bb75ac640 10 monclient(hunting): picked mon.noname-c con 0x7f1ba802b090 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:34.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.757+0000 7f1bb75ac640 10 monclient(hunting): picked mon.noname-b con 0x7f1ba802c7f0 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:34.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.757+0000 7f1bb75ac640 10 monclient(hunting): picked mon.noname-a con 0x7f1ba8006940 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:34.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.757+0000 7f1bb75ac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba8006dd0 con 0x7f1ba8006940 2024-03-29T10:46:34.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.757+0000 7f1bb75ac640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba802a430 con 0x7f1ba802b090 2024-03-29T10:46:34.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.757+0000 7f1bb75ac640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba802a5c0 con 0x7f1ba802c7f0 2024-03-29T10:46:34.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.757+0000 7f1bb75ac640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:34.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.757+0000 7f1bb75ac640 10 monclient(hunting): authenticate will time out at 1014.954877s 2024-03-29T10:46:34.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.761+0000 7f1bb5348640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1ba802b090 0x7f1ba80e8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:42991/0 (socket says 172.21.15.98:42991) 2024-03-29T10:46:34.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.761+0000 7f1bb5348640 1 -- 172.21.15.98:0/2324199108 learned_addr learned my addr 172.21.15.98:0/2324199108 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:34.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.761+0000 7f1ba7fff640 1 -- 172.21.15.98:0/2324199108 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970502155 0 0) 0x7f1ba802a430 con 0x7f1ba802b090 2024-03-29T10:46:34.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.761+0000 7f1ba7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:34.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.761+0000 7f1ba7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:34.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.761+0000 7f1ba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:34.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.761+0000 7f1ba7fff640 10 monclient(hunting): my global_id is 4294 2024-03-29T10:46:34.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.761+0000 7f1ba7fff640 1 -- 172.21.15.98:0/2324199108 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b9c003520 con 0x7f1ba802b090 2024-03-29T10:46:34.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.761+0000 7f1ba7fff640 1 -- 172.21.15.98:0/2324199108 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1216969550 0 0) 0x7f1b9c003520 con 0x7f1ba802b090 2024-03-29T10:46:34.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.761+0000 7f1ba7fff640 1 -- 172.21.15.98:0/2324199108 >> v1:172.21.15.98:6790/0 conn(0x7f1ba802c7f0 legacy=0x7f1ba8029c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:34.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.761+0000 7f1ba7fff640 1 -- 172.21.15.98:0/2324199108 >> v1:172.21.15.98:6789/0 conn(0x7f1ba8006940 legacy=0x7f1ba80f2fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:34.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:34.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 1 monclient: found mon.noname-c 2024-03-29T10:46:34.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:34.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:46:34.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 1 -- 172.21.15.98:0/2324199108 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba80ecc90 con 0x7f1ba802b090 2024-03-29T10:46:34.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:34.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:34.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:34.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 1 -- 172.21.15.98:0/2324199108 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f1bb004f730 con 0x7f1ba802b090 2024-03-29T10:46:34.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:34.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:34.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 10 monclient: dump: 2024-03-29T10:46:34.749 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:34.749 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:34.749 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:34.749 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:34.749 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:34.749 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:34.749 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:34.749 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:34.749 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:34.749 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:34.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1bb75ac640 5 monclient: authenticate success, global_id 4294 2024-03-29T10:46:34.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1bb75ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:34.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 1 -- 172.21.15.98:0/2324199108 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bb0075f10 con 0x7f1ba802b090 2024-03-29T10:46:34.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1ba7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:34.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1bb75ac640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:34.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.763+0000 7f1bb75ac640 4 set_mon_vals no callback set 2024-03-29T10:46:34.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.764+0000 7f1bb75ac640 10 monclient: shutdown 2024-03-29T10:46:34.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.764+0000 7f1bb75ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:34.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.764+0000 7f1bb75ac640 1 -- 172.21.15.98:0/2324199108 >> v1:172.21.15.189:6789/0 conn(0x7f1ba802b090 legacy=0x7f1ba80e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:34.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.764+0000 7f1ba7fff640 1 -- 172.21.15.98:0/2324199108 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f1bb00758b0 con 0x7f1ba802b090 2024-03-29T10:46:34.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.764+0000 7f1ba7fff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:46:34.750 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.764+0000 7f1bb75ac640 1 -- 172.21.15.98:0/2324199108 shutdown_connections 2024-03-29T10:46:34.751 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.764+0000 7f1bb75ac640 1 -- 172.21.15.98:0/2324199108 >> 172.21.15.98:0/2324199108 conn(0x7f1ba80058f0 msgr2=0x7f1ba8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:34.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.766+0000 7f1bb75ac640 1 -- 172.21.15.98:0/2324199108 shutdown_connections 2024-03-29T10:46:34.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.766+0000 7f1bb75ac640 1 -- 172.21.15.98:0/2324199108 wait complete. 2024-03-29T10:46:34.752 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.767+0000 7f1bb75ac640 10 monclient: build_initial_monmap 2024-03-29T10:46:34.753 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.767+0000 7f1bb75ac640 10 monclient: monmap: 2024-03-29T10:46:34.753 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:34.753 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:34.753 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:46:34.753 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:46:34.753 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:34.753 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:34.753 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:34.753 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:46:34.753 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:46:34.753 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:34.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.767+0000 7f1bb75ac640 1 librados: starting msgr at 2024-03-29T10:46:34.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.767+0000 7f1bb75ac640 1 librados: starting objecter 2024-03-29T10:46:34.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 1 Processor -- start 2024-03-29T10:46:34.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 1 -- start start 2024-03-29T10:46:34.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 1 librados: setting wanted keys 2024-03-29T10:46:34.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 1 librados: calling monclient init 2024-03-29T10:46:34.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 10 monclient: init 2024-03-29T10:46:34.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:34.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:34.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 10 monclient(hunting): picked mon.noname-c con 0x7f1ba8006940 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:34.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 10 monclient(hunting): picked mon.noname-b con 0x7f1ba802b090 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:34.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 10 monclient(hunting): picked mon.noname-a con 0x7f1ba802c7f0 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:34.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba80ec090 con 0x7f1ba802c7f0 2024-03-29T10:46:34.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba80ec300 con 0x7f1ba802b090 2024-03-29T10:46:34.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba80ea250 con 0x7f1ba8006940 2024-03-29T10:46:34.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:34.755 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.768+0000 7f1bb75ac640 10 monclient(hunting): authenticate will time out at 1014.965914s 2024-03-29T10:46:34.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.773+0000 7f1bb5b49640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1ba802c7f0 0x7f1ba8196850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35051/0 (socket says 172.21.15.98:35051) 2024-03-29T10:46:34.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.773+0000 7f1bb5b49640 1 -- 172.21.15.98:0/3342583152 learned_addr learned my addr 172.21.15.98:0/3342583152 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:34.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.775+0000 7f1ba5ffb640 10 client.?.objecter ms_handle_connect 0x7f1ba802c7f0 2024-03-29T10:46:34.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.775+0000 7f1ba5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:34.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.775+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833608644 0 0) 0x7f1ba80ec090 con 0x7f1ba802c7f0 2024-03-29T10:46:34.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.775+0000 7f1ba5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:34.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.775+0000 7f1ba5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:34.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.775+0000 7f1ba5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:34.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.775+0000 7f1ba5ffb640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:46:34.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.775+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b94004eb0 con 0x7f1ba802c7f0 2024-03-29T10:46:34.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.776+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 749742019 0 0) 0x7f1b94004eb0 con 0x7f1ba802c7f0 2024-03-29T10:46:34.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.776+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ba80ec090 con 0x7f1ba802c7f0 2024-03-29T10:46:34.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.776+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f1bac0018e0 con 0x7f1ba802c7f0 2024-03-29T10:46:34.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.776+0000 7f1ba5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:34.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.776+0000 7f1ba5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:34.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.776+0000 7f1ba5ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:34.760 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:34.760 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:34.760 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:34.760 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:34.760 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:34.761 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:34.761 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:34.761 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:34.761 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:34.761 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:34.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.776+0000 7f1ba5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:34.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.776+0000 7f1ba5ffb640 10 client.?.objecter ms_handle_connect 0x7f1ba802b090 2024-03-29T10:46:34.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.776+0000 7f1ba5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:34.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.777+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3590649468 0 0) 0x7f1ba80ec090 con 0x7f1ba802c7f0 2024-03-29T10:46:34.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.777+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 >> v1:172.21.15.98:6790/0 conn(0x7f1ba8006940 legacy=0x7f1ba80d99d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:34.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.777+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 >> v1:172.21.15.189:6789/0 conn(0x7f1ba802b090 legacy=0x7f1ba80ea770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:34.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.777+0000 7f1ba5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:34.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.777+0000 7f1ba5ffb640 1 monclient: found mon.a 2024-03-29T10:46:34.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.777+0000 7f1ba5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:34.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.777+0000 7f1ba5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:46:34.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.777+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba80334d0 con 0x7f1ba802c7f0 2024-03-29T10:46:34.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1ba5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:34.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1ba5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:34.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1ba5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:34.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1bb75ac640 5 monclient: authenticate success, global_id 4281 2024-03-29T10:46:34.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1bb75ac640 10 monclient: _renew_subs 2024-03-29T10:46:34.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1bb75ac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:46:34.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1bb75ac640 1 -- 172.21.15.98:0/3342583152 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba80da0e0 con 0x7f1ba802c7f0 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1bb75ac640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1bb75ac640 10 monclient: _renew_subs 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1bb75ac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1bb75ac640 1 -- 172.21.15.98:0/3342583152 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba80e2800 con 0x7f1ba802c7f0 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1bb75ac640 1 librados: init done 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bac001d30 con 0x7f1ba802c7f0 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1ba5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f1bac0038a0 con 0x7f1ba802c7f0 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1ba5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1ba5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1ba5ffb640 10 monclient: dump: 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:34.767 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:34.768 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:34.768 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:34.768 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:34.768 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:34.768 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:34.768 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:34.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.778+0000 7f1ba77fe640 4 set_mon_vals no callback set 2024-03-29T10:46:34.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.783+0000 7f1bb75ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:34.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.783+0000 7f1bb75ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:34.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.783+0000 7f1bb75ac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:46:34.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.783+0000 7f1bb75ac640 1 -- 172.21.15.98:0/3342583152 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b90005080 con 0x7f1ba802c7f0 2024-03-29T10:46:34.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.784+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f1bac003b00 con 0x7f1ba802c7f0 2024-03-29T10:46:34.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.784+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f1bac084a80 con 0x7f1ba802c7f0 2024-03-29T10:46:34.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.784+0000 7f1ba5ffb640 10 client.4281.objecter ms_dispatch 0x7f1bb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:34.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.784+0000 7f1ba5ffb640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:34.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.784+0000 7f1ba5ffb640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:34.770 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.784+0000 7f1ba5ffb640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-29T10:46:34.770 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.788+0000 7f1bb5348640 10 monclient: get_auth_request con 0x7f1b9406dc10 auth_method 0 2024-03-29T10:46:34.771 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.788+0000 7f1ba5ffb640 10 client.4281.objecter ms_handle_connect 0x7f1b9406dc10 2024-03-29T10:46:34.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.792+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1bac050360 con 0x7f1ba802c7f0 2024-03-29T10:46:34.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.792+0000 7f1ba5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:34.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:34.792+0000 7f1ba5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:35.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.118+0000 7f1bb75ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:35.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.118+0000 7f1bb75ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:35.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.118+0000 7f1bb75ac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:46:35.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.118+0000 7f1bb75ac640 1 -- 172.21.15.98:0/3342583152 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b90005330 con 0x7f1ba802c7f0 2024-03-29T10:46:35.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.119+0000 7f1ba5ffb640 1 -- 172.21.15.98:0/3342583152 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f1bac052290 con 0x7f1ba802c7f0 2024-03-29T10:46:35.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.119+0000 7f1ba5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:35.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.119+0000 7f1ba5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:35.104 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:46:35.104 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:35.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.125+0000 7f1bb75ac640 10 librados: watch_flush enter 2024-03-29T10:46:35.108 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.126+0000 7f1bb75ac640 10 librados: watch_flush exit 2024-03-29T10:46:35.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.126+0000 7f1bb75ac640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-29T10:46:35.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.126+0000 7f1bb75ac640 10 client.4281.objecter successfully canceled tick 2024-03-29T10:46:35.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.127+0000 7f1bb75ac640 1 -- 172.21.15.98:0/3342583152 >> v1:172.21.15.98:6816/1481799426 conn(0x7f1b9406dc10 legacy=0x7f1b940700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:35.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.127+0000 7f1bb75ac640 10 monclient: shutdown 2024-03-29T10:46:35.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.127+0000 7f1bb75ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:35.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.127+0000 7f1bb75ac640 1 -- 172.21.15.98:0/3342583152 >> v1:172.21.15.98:6789/0 conn(0x7f1ba802c7f0 legacy=0x7f1ba8196850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:35.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.127+0000 7f1bb75ac640 1 -- 172.21.15.98:0/3342583152 shutdown_connections 2024-03-29T10:46:35.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.127+0000 7f1bb75ac640 1 -- 172.21.15.98:0/3342583152 >> 172.21.15.98:0/3342583152 conn(0x7f1ba80058f0 msgr2=0x7f1ba80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:35.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.127+0000 7f1bb75ac640 1 -- 172.21.15.98:0/3342583152 shutdown_connections 2024-03-29T10:46:35.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.127+0000 7f1bb75ac640 1 -- 172.21.15.98:0/3342583152 wait complete. 2024-03-29T10:46:35.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:35.127+0000 7f1bb75ac640 1 librados: shutdown 2024-03-29T10:46:35.129 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:41.937 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:42.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.110+0000 7f2acf93e640 10 monclient: get_monmap_and_config 2024-03-29T10:46:42.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.110+0000 7f2acf93e640 10 monclient: build_initial_monmap 2024-03-29T10:46:42.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.110+0000 7f2acf93e640 10 monclient: monmap: 2024-03-29T10:46:42.094 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:42.094 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:42.094 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:46:42.111372+0000 2024-03-29T10:46:42.094 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:46:42.111372+0000 2024-03-29T10:46:42.094 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:42.094 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:42.094 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:42.095 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:46:42.095 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:46:42.095 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:42.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.111+0000 7f2acf93e640 1 Processor -- start 2024-03-29T10:46:42.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.112+0000 7f2acf93e640 1 -- start start 2024-03-29T10:46:42.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.112+0000 7f2acf93e640 10 monclient: init 2024-03-29T10:46:42.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.113+0000 7f2acf93e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:42.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.113+0000 7f2acf93e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:42.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.113+0000 7f2acf93e640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac80a94e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:42.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.113+0000 7f2acf93e640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac813ae90 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:42.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.113+0000 7f2acf93e640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac80aae40 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:42.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.113+0000 7f2acf93e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac80ac430 con 0x7f2ac80aae40 2024-03-29T10:46:42.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.113+0000 7f2acf93e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac80ac960 con 0x7f2ac80a94e0 2024-03-29T10:46:42.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.113+0000 7f2acf93e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac813baf0 con 0x7f2ac813ae90 2024-03-29T10:46:42.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.113+0000 7f2acf93e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:42.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.113+0000 7f2acf93e640 10 monclient(hunting): authenticate will time out at 1022.310446s 2024-03-29T10:46:42.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.116+0000 7f2acdedb640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2ac80aae40 0x7f2ac80ab270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33779/0 (socket says 172.21.15.98:33779) 2024-03-29T10:46:42.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.116+0000 7f2acdedb640 1 -- 172.21.15.98:0/2835567471 learned_addr learned my addr 172.21.15.98:0/2835567471 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:42.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.120+0000 7f2abffff640 1 -- 172.21.15.98:0/2835567471 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307900544 0 0) 0x7f2ac80ac430 con 0x7f2ac80aae40 2024-03-29T10:46:42.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.120+0000 7f2abffff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:42.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.120+0000 7f2abffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:42.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.120+0000 7f2abffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:42.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.120+0000 7f2abffff640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:46:42.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.120+0000 7f2abffff640 1 -- 172.21.15.98:0/2835567471 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab4003520 con 0x7f2ac80aae40 2024-03-29T10:46:42.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.120+0000 7f2abffff640 1 -- 172.21.15.98:0/2835567471 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954585744 0 0) 0x7f2ac80ac960 con 0x7f2ac80a94e0 2024-03-29T10:46:42.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.120+0000 7f2abffff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:42.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.120+0000 7f2abffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:42.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.120+0000 7f2abffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:42.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.120+0000 7f2abffff640 10 monclient(hunting): my global_id is 4300 2024-03-29T10:46:42.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.120+0000 7f2abffff640 1 -- 172.21.15.98:0/2835567471 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac80ac430 con 0x7f2ac80a94e0 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 1 -- 172.21.15.98:0/2835567471 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 9130186 0 0) 0x7f2ac80ac430 con 0x7f2ac80a94e0 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 1 -- 172.21.15.98:0/2835567471 >> v1:172.21.15.98:6790/0 conn(0x7f2ac813ae90 legacy=0x7f2ac813b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 1 -- 172.21.15.98:0/2835567471 >> v1:172.21.15.98:6789/0 conn(0x7f2ac80aae40 legacy=0x7f2ac80ab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 1 monclient: found mon.noname-c 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 1 -- 172.21.15.98:0/2835567471 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80afe70 con 0x7f2ac80a94e0 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 10 monclient: _finish_auth 0 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2acf93e640 5 monclient: authenticate success, global_id 4300 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2acf93e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:42.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 1 -- 172.21.15.98:0/2835567471 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2ac40015f0 con 0x7f2ac80a94e0 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 10 monclient: dump: 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:42.105 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2acf93e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 1 -- 172.21.15.98:0/2835567471 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ac4001810 con 0x7f2ac80a94e0 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2abffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2acf93e640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.121+0000 7f2acf93e640 4 set_mon_vals no callback set 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.122+0000 7f2abffff640 1 -- 172.21.15.98:0/2835567471 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2ac4001ba0 con 0x7f2ac80a94e0 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.122+0000 7f2abffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.122+0000 7f2abffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.122+0000 7f2abffff640 10 monclient: dump: 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:42.106 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:42.107 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:42.107 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:42.107 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:42.107 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.122+0000 7f2acf93e640 10 monclient: shutdown 2024-03-29T10:46:42.107 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.122+0000 7f2acf93e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:42.107 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.122+0000 7f2acf93e640 1 -- 172.21.15.98:0/2835567471 >> v1:172.21.15.189:6789/0 conn(0x7f2ac80a94e0 legacy=0x7f2ac80a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:42.107 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.124+0000 7f2acf93e640 1 -- 172.21.15.98:0/2835567471 shutdown_connections 2024-03-29T10:46:42.107 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.124+0000 7f2acf93e640 1 -- 172.21.15.98:0/2835567471 >> 172.21.15.98:0/2835567471 conn(0x7f2ac80b3a00 msgr2=0x7f2ac80b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:42.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.125+0000 7f2acf93e640 1 -- 172.21.15.98:0/2835567471 shutdown_connections 2024-03-29T10:46:42.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.128+0000 7f2acf93e640 1 -- 172.21.15.98:0/2835567471 wait complete. 2024-03-29T10:46:42.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.129+0000 7f2acf93e640 10 monclient: build_initial_monmap 2024-03-29T10:46:42.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.129+0000 7f2acf93e640 10 monclient: monmap: 2024-03-29T10:46:42.117 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:42.117 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:42.118 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:46:42.118 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:46:42.118 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:42.118 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:42.118 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:42.118 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:46:42.118 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:46:42.118 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:42.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.129+0000 7f2acf93e640 1 librados: starting msgr at 2024-03-29T10:46:42.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.129+0000 7f2acf93e640 1 librados: starting objecter 2024-03-29T10:46:42.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.129+0000 7f2acf93e640 1 Processor -- start 2024-03-29T10:46:42.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.130+0000 7f2acf93e640 1 -- start start 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.130+0000 7f2acf93e640 1 librados: setting wanted keys 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.130+0000 7f2acf93e640 1 librados: calling monclient init 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.130+0000 7f2acf93e640 10 monclient: init 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.131+0000 7f2acf93e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.131+0000 7f2acf93e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.131+0000 7f2acf93e640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac80a94e0 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.131+0000 7f2acf93e640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac80aae40 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.131+0000 7f2acf93e640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac813ae90 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.131+0000 7f2acf93e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac813e060 con 0x7f2ac80a94e0 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.131+0000 7f2acf93e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac813e340 con 0x7f2ac80aae40 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.131+0000 7f2acf93e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac8136960 con 0x7f2ac813ae90 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.131+0000 7f2acf93e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.131+0000 7f2acf93e640 10 monclient(hunting): authenticate will time out at 1022.328090s 2024-03-29T10:46:42.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.135+0000 7f2acced9640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2ac80aae40 0x7f2ac81390c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:40147/0 (socket says 172.21.15.98:40147) 2024-03-29T10:46:42.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.135+0000 7f2acced9640 1 -- 172.21.15.98:0/2772638905 learned_addr learned my addr 172.21.15.98:0/2772638905 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:42.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.136+0000 7f2abdffb640 10 client.?.objecter ms_handle_connect 0x7f2ac80aae40 2024-03-29T10:46:42.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.136+0000 7f2abdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:42.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.140+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693295789 0 0) 0x7f2ac813e340 con 0x7f2ac80aae40 2024-03-29T10:46:42.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.140+0000 7f2abdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:42.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.140+0000 7f2abdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:42.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.140+0000 7f2abdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:42.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.140+0000 7f2abdffb640 10 monclient(hunting): my global_id is 4303 2024-03-29T10:46:42.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.140+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aac004eb0 con 0x7f2ac80aae40 2024-03-29T10:46:42.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 10 client.?.objecter ms_handle_connect 0x7f2ac813ae90 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025484113 0 0) 0x7f2ac8136960 con 0x7f2ac813ae90 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac813e340 con 0x7f2ac813ae90 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2096764858 0 0) 0x7f2aac004eb0 con 0x7f2ac80aae40 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ac8136960 con 0x7f2ac80aae40 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2ab80017d0 con 0x7f2ac80aae40 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 10 monclient(hunting): dump: 2024-03-29T10:46:42.124 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:42.125 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:42.125 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:42.125 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:42.125 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:42.125 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:42.125 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:42.125 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:42.125 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:42.125 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:42.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.141+0000 7f2abdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:42.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2916030213 0 0) 0x7f2ac8136960 con 0x7f2ac80aae40 2024-03-29T10:46:42.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 >> v1:172.21.15.98:6790/0 conn(0x7f2ac813ae90 legacy=0x7f2ac81dd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 >> v1:172.21.15.98:6789/0 conn(0x7f2ac80a94e0 legacy=0x7f2ac81389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2abdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2abdffb640 1 monclient: found mon.b 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2abdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2abdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac8136580 con 0x7f2ac80aae40 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2abdffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2abdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2abdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2acf93e640 5 monclient: authenticate success, global_id 4303 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2acf93e640 10 monclient: _renew_subs 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2acf93e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.142+0000 7f2acf93e640 1 -- 172.21.15.98:0/2772638905 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac8133610 con 0x7f2ac80aae40 2024-03-29T10:46:42.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.143+0000 7f2acf93e640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:42.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.143+0000 7f2acf93e640 10 monclient: _renew_subs 2024-03-29T10:46:42.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.143+0000 7f2acf93e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:42.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.143+0000 7f2acf93e640 1 -- 172.21.15.98:0/2772638905 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac813ecb0 con 0x7f2ac80aae40 2024-03-29T10:46:42.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.143+0000 7f2acf93e640 1 librados: init done 2024-03-29T10:46:42.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.146+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ab8001ce0 con 0x7f2ac80aae40 2024-03-29T10:46:42.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.146+0000 7f2abdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:42.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.146+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2ab8004a00 con 0x7f2ac80aae40 2024-03-29T10:46:42.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.146+0000 7f2abdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:42.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.146+0000 7f2abdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:42.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.146+0000 7f2abdffb640 10 monclient: dump: 2024-03-29T10:46:42.133 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:42.133 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:42.133 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:42.133 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:42.133 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:42.133 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:42.133 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:42.133 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:42.134 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:42.134 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:42.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.146+0000 7f2abeffd640 4 set_mon_vals no callback set 2024-03-29T10:46:42.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.147+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f2ab8004ba0 con 0x7f2ac80aae40 2024-03-29T10:46:42.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.147+0000 7f2acf93e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:42.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.147+0000 7f2acf93e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:42.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.147+0000 7f2acf93e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:42.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.147+0000 7f2acf93e640 1 -- 172.21.15.98:0/2772638905 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa8005080 con 0x7f2ac80aae40 2024-03-29T10:46:42.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.148+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f2ab8084a20 con 0x7f2ac80aae40 2024-03-29T10:46:42.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.148+0000 7f2abdffb640 10 client.4303.objecter ms_dispatch 0x7f2ac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:42.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.148+0000 7f2abdffb640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:42.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.148+0000 7f2abdffb640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:42.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.148+0000 7f2abdffb640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-29T10:46:42.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.148+0000 7f2acd6da640 10 monclient: get_auth_request con 0x7f2aac06dd00 auth_method 0 2024-03-29T10:46:42.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.148+0000 7f2abdffb640 10 client.4303.objecter ms_handle_connect 0x7f2aac06dd00 2024-03-29T10:46:42.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.155+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ab8050460 con 0x7f2ac80aae40 2024-03-29T10:46:42.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.155+0000 7f2abdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:42.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.155+0000 7f2abdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:42.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.481+0000 7f2acf93e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:42.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.481+0000 7f2acf93e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:42.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.481+0000 7f2acf93e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:42.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.481+0000 7f2acf93e640 1 -- 172.21.15.98:0/2772638905 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2aa8005330 con 0x7f2ac80aae40 2024-03-29T10:46:42.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.482+0000 7f2abdffb640 1 -- 172.21.15.98:0/2772638905 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f2ab8052390 con 0x7f2ac80aae40 2024-03-29T10:46:42.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.482+0000 7f2abdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:42.468 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:46:42.468 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:42.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.482+0000 7f2abdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:42.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.486+0000 7f2acf93e640 10 librados: watch_flush enter 2024-03-29T10:46:42.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.486+0000 7f2acf93e640 10 librados: watch_flush exit 2024-03-29T10:46:42.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.486+0000 7f2acf93e640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-29T10:46:42.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.486+0000 7f2acf93e640 10 client.4303.objecter successfully canceled tick 2024-03-29T10:46:42.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.488+0000 7f2acf93e640 1 -- 172.21.15.98:0/2772638905 >> v1:172.21.15.98:6816/1481799426 conn(0x7f2aac06dd00 legacy=0x7f2aac0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:42.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.488+0000 7f2acf93e640 10 monclient: shutdown 2024-03-29T10:46:42.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.488+0000 7f2acf93e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:42.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.488+0000 7f2acf93e640 1 -- 172.21.15.98:0/2772638905 >> v1:172.21.15.189:6789/0 conn(0x7f2ac80aae40 legacy=0x7f2ac81390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:42.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.488+0000 7f2acf93e640 1 -- 172.21.15.98:0/2772638905 shutdown_connections 2024-03-29T10:46:42.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.488+0000 7f2acf93e640 1 -- 172.21.15.98:0/2772638905 >> 172.21.15.98:0/2772638905 conn(0x7f2ac80b3a00 msgr2=0x7f2ac80b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:42.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.490+0000 7f2acf93e640 1 -- 172.21.15.98:0/2772638905 shutdown_connections 2024-03-29T10:46:42.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.492+0000 7f2acf93e640 1 -- 172.21.15.98:0/2772638905 wait complete. 2024-03-29T10:46:42.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:42.492+0000 7f2acf93e640 1 librados: shutdown 2024-03-29T10:46:42.497 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:49.304 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:49.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.483+0000 7f37d21e8640 10 monclient: get_monmap_and_config 2024-03-29T10:46:49.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.483+0000 7f37d21e8640 10 monclient: build_initial_monmap 2024-03-29T10:46:49.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.483+0000 7f37d21e8640 10 monclient: monmap: 2024-03-29T10:46:49.465 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:49.466 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:49.466 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:46:49.483931+0000 2024-03-29T10:46:49.466 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:46:49.483931+0000 2024-03-29T10:46:49.466 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:49.466 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:49.466 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:49.466 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:46:49.466 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:46:49.466 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:49.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.484+0000 7f37d21e8640 1 Processor -- start 2024-03-29T10:46:49.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.484+0000 7f37d21e8640 1 -- start start 2024-03-29T10:46:49.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.484+0000 7f37d21e8640 10 monclient: init 2024-03-29T10:46:49.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.485+0000 7f37d21e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:49.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.485+0000 7f37d21e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:49.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.485+0000 7f37d21e8640 10 monclient(hunting): picked mon.noname-c con 0x7f37cc0b2800 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:49.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.485+0000 7f37d21e8640 10 monclient(hunting): picked mon.noname-b con 0x7f37cc142b20 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:49.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.485+0000 7f37d21e8640 10 monclient(hunting): picked mon.noname-a con 0x7f37cc13c040 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:49.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.485+0000 7f37d21e8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc0aad10 con 0x7f37cc13c040 2024-03-29T10:46:49.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.485+0000 7f37d21e8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc141f30 con 0x7f37cc0b2800 2024-03-29T10:46:49.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.485+0000 7f37d21e8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc0aaf60 con 0x7f37cc142b20 2024-03-29T10:46:49.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.485+0000 7f37d21e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:49.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.485+0000 7f37d21e8640 10 monclient(hunting): authenticate will time out at 1029.682458s 2024-03-29T10:46:49.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.488+0000 7f37d19e7640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f37cc13c040 0x7f37cc14d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53972/0 (socket says 172.21.15.98:53972) 2024-03-29T10:46:49.472 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.488+0000 7f37d19e7640 1 -- 172.21.15.98:0/152997704 learned_addr learned my addr 172.21.15.98:0/152997704 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:49.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.490+0000 7f37c3fff640 1 -- 172.21.15.98:0/152997704 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171499118 0 0) 0x7f37cc141f30 con 0x7f37cc0b2800 2024-03-29T10:46:49.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.490+0000 7f37c3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:49.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.490+0000 7f37c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:49.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.490+0000 7f37c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:49.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.490+0000 7f37c3fff640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:46:49.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.490+0000 7f37c3fff640 1 -- 172.21.15.98:0/152997704 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b8003520 con 0x7f37cc0b2800 2024-03-29T10:46:49.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.490+0000 7f37c3fff640 1 -- 172.21.15.98:0/152997704 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 640896940 0 0) 0x7f37b8003520 con 0x7f37cc0b2800 2024-03-29T10:46:49.473 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.490+0000 7f37c3fff640 1 -- 172.21.15.98:0/152997704 >> v1:172.21.15.98:6790/0 conn(0x7f37cc142b20 legacy=0x7f37cc13b930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:49.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37c3fff640 1 -- 172.21.15.98:0/152997704 >> v1:172.21.15.98:6789/0 conn(0x7f37cc13c040 legacy=0x7f37cc14d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:49.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37c3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:49.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37c3fff640 1 monclient: found mon.noname-c 2024-03-29T10:46:49.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:49.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37c3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:46:49.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37c3fff640 1 -- 172.21.15.98:0/152997704 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc0b3900 con 0x7f37cc0b2800 2024-03-29T10:46:49.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37c3fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:49.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37c3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:49.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:49.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37d21e8640 5 monclient: authenticate success, global_id 4306 2024-03-29T10:46:49.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37d21e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:49.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37c3fff640 1 -- 172.21.15.98:0/152997704 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f37c40019f0 con 0x7f37cc0b2800 2024-03-29T10:46:49.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:49.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:49.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37c3fff640 10 monclient: dump: 2024-03-29T10:46:49.476 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:49.476 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:49.477 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:49.477 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:49.477 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:49.477 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:49.477 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:49.477 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:49.477 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:49.477 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:49.477 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.492+0000 7f37d21e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:49.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.493+0000 7f37c3fff640 1 -- 172.21.15.98:0/152997704 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37c4001bc0 con 0x7f37cc0b2800 2024-03-29T10:46:49.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.493+0000 7f37c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:49.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.493+0000 7f37d21e8640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:49.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.493+0000 7f37d21e8640 4 set_mon_vals no callback set 2024-03-29T10:46:49.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.493+0000 7f37c3fff640 1 -- 172.21.15.98:0/152997704 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f37c4001fd0 con 0x7f37cc0b2800 2024-03-29T10:46:49.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.493+0000 7f37c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:49.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.493+0000 7f37c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:49.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.493+0000 7f37c3fff640 10 monclient: dump: 2024-03-29T10:46:49.481 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:49.481 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:49.481 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:49.481 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:49.481 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.494+0000 7f37d21e8640 10 monclient: shutdown 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.494+0000 7f37d21e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.494+0000 7f37d21e8640 1 -- 172.21.15.98:0/152997704 >> v1:172.21.15.189:6789/0 conn(0x7f37cc0b2800 legacy=0x7f37cc142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.494+0000 7f37d21e8640 1 -- 172.21.15.98:0/152997704 shutdown_connections 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.494+0000 7f37d21e8640 1 -- 172.21.15.98:0/152997704 >> 172.21.15.98:0/152997704 conn(0x7f37cc06d0e0 msgr2=0x7f37cc066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.494+0000 7f37d21e8640 1 -- 172.21.15.98:0/152997704 shutdown_connections 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.494+0000 7f37d21e8640 1 -- 172.21.15.98:0/152997704 wait complete. 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.495+0000 7f37d21e8640 10 monclient: build_initial_monmap 2024-03-29T10:46:49.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.495+0000 7f37d21e8640 10 monclient: monmap: 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.496+0000 7f37d21e8640 1 librados: starting msgr at 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.496+0000 7f37d21e8640 1 librados: starting objecter 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.496+0000 7f37d21e8640 1 Processor -- start 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.496+0000 7f37d21e8640 1 -- start start 2024-03-29T10:46:49.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.496+0000 7f37d21e8640 1 librados: setting wanted keys 2024-03-29T10:46:49.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.496+0000 7f37d21e8640 1 librados: calling monclient init 2024-03-29T10:46:49.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.496+0000 7f37d21e8640 10 monclient: init 2024-03-29T10:46:49.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.497+0000 7f37d21e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:49.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.497+0000 7f37d21e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:49.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.497+0000 7f37d21e8640 10 monclient(hunting): picked mon.noname-a con 0x7f37cc0b2800 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:49.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.497+0000 7f37d21e8640 10 monclient(hunting): picked mon.noname-c con 0x7f37cc13c040 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:49.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.497+0000 7f37d21e8640 10 monclient(hunting): picked mon.noname-b con 0x7f37cc142b20 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:49.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.497+0000 7f37d21e8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc1457b0 con 0x7f37cc0b2800 2024-03-29T10:46:49.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.497+0000 7f37d21e8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc1dc170 con 0x7f37cc142b20 2024-03-29T10:46:49.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.497+0000 7f37d21e8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc145480 con 0x7f37cc13c040 2024-03-29T10:46:49.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.497+0000 7f37d21e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:49.484 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.497+0000 7f37d21e8640 10 monclient(hunting): authenticate will time out at 1029.694873s 2024-03-29T10:46:49.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.501+0000 7f37d19e7640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f37cc142b20 0x7f37cc1dba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:48683/0 (socket says 172.21.15.98:48683) 2024-03-29T10:46:49.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.501+0000 7f37d19e7640 1 -- 172.21.15.98:0/3539039564 learned_addr learned my addr 172.21.15.98:0/3539039564 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:49.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.501+0000 7f37c1ffb640 10 client.?.objecter ms_handle_connect 0x7f37cc142b20 2024-03-29T10:46:49.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.501+0000 7f37c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:49.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 10 client.?.objecter ms_handle_connect 0x7f37cc0b2800 2024-03-29T10:46:49.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:49.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976426070 0 0) 0x7f37cc1dc170 con 0x7f37cc142b20 2024-03-29T10:46:49.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:49.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:49.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:49.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:46:49.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b0004eb0 con 0x7f37cc142b20 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968906068 0 0) 0x7f37cc1457b0 con 0x7f37cc0b2800 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.502+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37cc1dc170 con 0x7f37cc0b2800 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.503+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3400740912 0 0) 0x7f37b0004eb0 con 0x7f37cc142b20 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.503+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37cc1457b0 con 0x7f37cc142b20 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.503+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 632798861 0 0) 0x7f37cc1dc170 con 0x7f37cc0b2800 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.503+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37b0004eb0 con 0x7f37cc0b2800 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.504+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f37bc001930 con 0x7f37cc142b20 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.504+0000 7f37c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.504+0000 7f37c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:49.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.504+0000 7f37c1ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:49.493 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:49.493 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:49.493 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:49.493 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:49.493 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:49.493 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:49.493 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:49.493 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:49.493 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:49.493 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:49.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.504+0000 7f37c1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:49.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.504+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f37c4001ed0 con 0x7f37cc0b2800 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.504+0000 7f37c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.504+0000 7f37c1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.504+0000 7f37c1ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:49.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.504+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3058400845 0 0) 0x7f37cc1457b0 con 0x7f37cc142b20 2024-03-29T10:46:49.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.504+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 >> v1:172.21.15.98:6790/0 conn(0x7f37cc13c040 legacy=0x7f37cc1db350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:49.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.504+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 >> v1:172.21.15.98:6789/0 conn(0x7f37cc0b2800 legacy=0x7f37cc1de770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:49.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.504+0000 7f37c1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:49.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37c1ffb640 1 monclient: found mon.b 2024-03-29T10:46:49.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:49.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37c1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:49.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc06e5f0 con 0x7f37cc142b20 2024-03-29T10:46:49.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37c1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:49.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37c1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:49.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:49.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37d21e8640 5 monclient: authenticate success, global_id 4309 2024-03-29T10:46:49.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37d21e8640 10 monclient: _renew_subs 2024-03-29T10:46:49.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37d21e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37d21e8640 1 -- 172.21.15.98:0/3539039564 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37cc06e8d0 con 0x7f37cc142b20 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37d21e8640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37d21e8640 10 monclient: _renew_subs 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37d21e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37d21e8640 1 -- 172.21.15.98:0/3539039564 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37cc1df060 con 0x7f37cc142b20 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37d21e8640 1 librados: init done 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37bc0047e0 con 0x7f37cc142b20 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.505+0000 7f37c37fe640 4 set_mon_vals no callback set 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.506+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f37bc004950 con 0x7f37cc142b20 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.506+0000 7f37c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.506+0000 7f37c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.506+0000 7f37c1ffb640 10 monclient: dump: 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:49.496 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:49.497 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:49.497 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:49.497 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:49.497 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:49.497 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:49.497 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:49.497 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:49.497 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:49.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.507+0000 7f37d21e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:49.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.507+0000 7f37d21e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:49.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.507+0000 7f37d21e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:49.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.507+0000 7f37d21e8640 1 -- 172.21.15.98:0/3539039564 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ac005080 con 0x7f37cc142b20 2024-03-29T10:46:49.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.508+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f37bc01a920 con 0x7f37cc142b20 2024-03-29T10:46:49.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.509+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f37bc084800 con 0x7f37cc142b20 2024-03-29T10:46:49.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.509+0000 7f37c1ffb640 10 client.4309.objecter ms_dispatch 0x7f37cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:49.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.509+0000 7f37c1ffb640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:49.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.509+0000 7f37c1ffb640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:49.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.509+0000 7f37c1ffb640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-29T10:46:49.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.511+0000 7f37d11e6640 10 monclient: get_auth_request con 0x7f37b006de20 auth_method 0 2024-03-29T10:46:49.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.513+0000 7f37c1ffb640 10 client.4309.objecter ms_handle_connect 0x7f37b006de20 2024-03-29T10:46:49.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.516+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37bc0500e0 con 0x7f37cc142b20 2024-03-29T10:46:49.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.516+0000 7f37c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:49.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.516+0000 7f37c1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:49.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.874+0000 7f37d21e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:49.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.874+0000 7f37d21e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:49.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.874+0000 7f37d21e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:49.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.874+0000 7f37d21e8640 1 -- 172.21.15.98:0/3539039564 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37ac005330 con 0x7f37cc142b20 2024-03-29T10:46:49.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.875+0000 7f37c1ffb640 1 -- 172.21.15.98:0/3539039564 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f37bc052010 con 0x7f37cc142b20 2024-03-29T10:46:49.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.875+0000 7f37c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:49.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.875+0000 7f37c1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:49.858 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:46:49.858 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:49.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.878+0000 7f37d21e8640 10 librados: watch_flush enter 2024-03-29T10:46:49.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.878+0000 7f37d21e8640 10 librados: watch_flush exit 2024-03-29T10:46:49.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.878+0000 7f37d21e8640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-29T10:46:49.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.878+0000 7f37d21e8640 10 client.4309.objecter successfully canceled tick 2024-03-29T10:46:49.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.879+0000 7f37d21e8640 1 -- 172.21.15.98:0/3539039564 >> v1:172.21.15.98:6816/1481799426 conn(0x7f37b006de20 legacy=0x7f37b00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:49.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.879+0000 7f37d21e8640 10 monclient: shutdown 2024-03-29T10:46:49.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.879+0000 7f37d21e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:49.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.879+0000 7f37d21e8640 1 -- 172.21.15.98:0/3539039564 >> v1:172.21.15.189:6789/0 conn(0x7f37cc142b20 legacy=0x7f37cc1dba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:49.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.879+0000 7f37d21e8640 1 -- 172.21.15.98:0/3539039564 shutdown_connections 2024-03-29T10:46:49.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.879+0000 7f37d21e8640 1 -- 172.21.15.98:0/3539039564 >> 172.21.15.98:0/3539039564 conn(0x7f37cc06d0e0 msgr2=0x7f37cc066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:49.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.879+0000 7f37d21e8640 1 -- 172.21.15.98:0/3539039564 shutdown_connections 2024-03-29T10:46:49.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.879+0000 7f37d21e8640 1 -- 172.21.15.98:0/3539039564 wait complete. 2024-03-29T10:46:49.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:49.879+0000 7f37d21e8640 1 librados: shutdown 2024-03-29T10:46:49.882 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:56.689 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:56.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.862+0000 7f67ce6b3640 10 monclient: get_monmap_and_config 2024-03-29T10:46:56.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.862+0000 7f67ce6b3640 10 monclient: build_initial_monmap 2024-03-29T10:46:56.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.862+0000 7f67ce6b3640 10 monclient: monmap: 2024-03-29T10:46:56.845 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:56.845 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:56.845 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:46:56.863143+0000 2024-03-29T10:46:56.845 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:46:56.863143+0000 2024-03-29T10:46:56.845 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:56.845 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:56.845 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:56.845 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:46:56.845 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:46:56.845 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:56.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.863+0000 7f67ce6b3640 1 Processor -- start 2024-03-29T10:46:56.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.863+0000 7f67ce6b3640 1 -- start start 2024-03-29T10:46:56.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.863+0000 7f67ce6b3640 10 monclient: init 2024-03-29T10:46:56.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.864+0000 7f67ce6b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:56.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.864+0000 7f67ce6b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:56.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.864+0000 7f67ce6b3640 10 monclient(hunting): picked mon.noname-a con 0x7f67c80b0770 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:56.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.864+0000 7f67c7fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f67c80b0770 0x7f67c8072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47098/0 (socket says 172.21.15.98:47098) 2024-03-29T10:46:56.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.864+0000 7f67c7fff640 1 -- 172.21.15.98:0/3678656961 learned_addr learned my addr 172.21.15.98:0/3678656961 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:56.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.864+0000 7f67ce6b3640 10 monclient(hunting): picked mon.noname-c con 0x7f67c80ad850 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:56.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.865+0000 7f67ce6b3640 10 monclient(hunting): picked mon.noname-b con 0x7f67c813c1b0 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:56.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.865+0000 7f67ce6b3640 1 -- 172.21.15.98:0/3678656961 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c80ac8e0 con 0x7f67c80b0770 2024-03-29T10:46:56.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.865+0000 7f67ce6b3640 1 -- 172.21.15.98:0/3678656961 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c80ad330 con 0x7f67c80ad850 2024-03-29T10:46:56.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.865+0000 7f67ce6b3640 1 -- 172.21.15.98:0/3678656961 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c80acfe0 con 0x7f67c813c1b0 2024-03-29T10:46:56.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.865+0000 7f67ce6b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:56.848 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.865+0000 7f67ce6b3640 10 monclient(hunting): authenticate will time out at 1037.062308s 2024-03-29T10:46:56.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 1 -- 172.21.15.98:0/3678656961 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120340596 0 0) 0x7f67c80ac8e0 con 0x7f67c80b0770 2024-03-29T10:46:56.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:56.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:56.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:56.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient(hunting): my global_id is 4293 2024-03-29T10:46:56.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 1 -- 172.21.15.98:0/3678656961 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b0003520 con 0x7f67c80b0770 2024-03-29T10:46:56.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 1 -- 172.21.15.98:0/3678656961 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38458170 0 0) 0x7f67c80ad330 con 0x7f67c80ad850 2024-03-29T10:46:56.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:56.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:56.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:56.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:46:56.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 1 -- 172.21.15.98:0/3678656961 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c80ac8e0 con 0x7f67c80ad850 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 1 -- 172.21.15.98:0/3678656961 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182681852 0 0) 0x7f67c80acfe0 con 0x7f67c813c1b0 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 1 -- 172.21.15.98:0/3678656961 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c80ad330 con 0x7f67c813c1b0 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 1 -- 172.21.15.98:0/3678656961 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2534423058 0 0) 0x7f67c80ad330 con 0x7f67c813c1b0 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 1 -- 172.21.15.98:0/3678656961 >> v1:172.21.15.189:6789/0 conn(0x7f67c80ad850 legacy=0x7f67c80aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 1 -- 172.21.15.98:0/3678656961 >> v1:172.21.15.98:6789/0 conn(0x7f67c80b0770 legacy=0x7f67c8072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 1 monclient: found mon.noname-b 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:46:56.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 1 -- 172.21.15.98:0/3678656961 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c80ab1f0 con 0x7f67c813c1b0 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 1 -- 172.21.15.98:0/3678656961 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f67c00018b0 con 0x7f67c813c1b0 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67c6ffd640 10 monclient: dump: 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:56.853 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67ce6b3640 5 monclient: authenticate success, global_id 4307 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.866+0000 7f67ce6b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.867+0000 7f67c6ffd640 1 -- 172.21.15.98:0/3678656961 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67c0001a90 con 0x7f67c813c1b0 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.867+0000 7f67c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.867+0000 7f67ce6b3640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.867+0000 7f67c6ffd640 1 -- 172.21.15.98:0/3678656961 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f67c00018b0 con 0x7f67c813c1b0 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.867+0000 7f67ce6b3640 4 set_mon_vals no callback set 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.867+0000 7f67c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.867+0000 7f67c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.867+0000 7f67c6ffd640 10 monclient: dump: 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:56.854 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:56.855 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:56.855 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:56.855 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:56.855 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:56.855 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:56.855 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:56.855 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:56.855 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:56.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.868+0000 7f67ce6b3640 10 monclient: shutdown 2024-03-29T10:46:56.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.868+0000 7f67ce6b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:56.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.868+0000 7f67ce6b3640 1 -- 172.21.15.98:0/3678656961 >> v1:172.21.15.98:6790/0 conn(0x7f67c813c1b0 legacy=0x7f67c814a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:56.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.868+0000 7f67ce6b3640 1 -- 172.21.15.98:0/3678656961 shutdown_connections 2024-03-29T10:46:56.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.868+0000 7f67ce6b3640 1 -- 172.21.15.98:0/3678656961 >> 172.21.15.98:0/3678656961 conn(0x7f67c806b660 msgr2=0x7f67c806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:56.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.868+0000 7f67ce6b3640 1 -- 172.21.15.98:0/3678656961 shutdown_connections 2024-03-29T10:46:56.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.869+0000 7f67ce6b3640 1 -- 172.21.15.98:0/3678656961 wait complete. 2024-03-29T10:46:56.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.870+0000 7f67ce6b3640 10 monclient: build_initial_monmap 2024-03-29T10:46:56.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.870+0000 7f67ce6b3640 10 monclient: monmap: 2024-03-29T10:46:56.860 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:46:56.860 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:56.860 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.870+0000 7f67ce6b3640 1 librados: starting msgr at 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.870+0000 7f67ce6b3640 1 librados: starting objecter 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.870+0000 7f67ce6b3640 1 Processor -- start 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.871+0000 7f67ce6b3640 1 -- start start 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.871+0000 7f67ce6b3640 1 librados: setting wanted keys 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.871+0000 7f67ce6b3640 1 librados: calling monclient init 2024-03-29T10:46:56.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.871+0000 7f67ce6b3640 10 monclient: init 2024-03-29T10:46:56.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67ce6b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:56.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67ce6b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:56.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67ce6b3640 10 monclient(hunting): picked mon.noname-a con 0x7f67c80ad850 addr v1:172.21.15.98:6789/0 2024-03-29T10:46:56.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67ce6b3640 10 monclient(hunting): picked mon.noname-c con 0x7f67c80b0770 addr v1:172.21.15.98:6790/0 2024-03-29T10:46:56.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67c7fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f67c80ad850 0x7f67c81d3ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47440/0 (socket says 172.21.15.98:47440) 2024-03-29T10:46:56.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67c7fff640 1 -- 172.21.15.98:0/4283351168 learned_addr learned my addr 172.21.15.98:0/4283351168 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:46:56.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67ce6b3640 10 monclient(hunting): picked mon.noname-b con 0x7f67c813c1b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:46:56.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67c4ff9640 10 client.?.objecter ms_handle_connect 0x7f67c80ad850 2024-03-29T10:46:56.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:56.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67ce6b3640 1 -- 172.21.15.98:0/4283351168 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c8135250 con 0x7f67c80ad850 2024-03-29T10:46:56.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67ce6b3640 1 -- 172.21.15.98:0/4283351168 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c806fd30 con 0x7f67c813c1b0 2024-03-29T10:46:56.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67ce6b3640 1 -- 172.21.15.98:0/4283351168 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c81d19e0 con 0x7f67c80b0770 2024-03-29T10:46:56.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67ce6b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:56.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.872+0000 7f67ce6b3640 10 monclient(hunting): authenticate will time out at 1037.069749s 2024-03-29T10:46:56.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 10 client.?.objecter ms_handle_connect 0x7f67c813c1b0 2024-03-29T10:46:56.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:56.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303316713 0 0) 0x7f67c8135250 con 0x7f67c80ad850 2024-03-29T10:46:56.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:56.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:56.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:56.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 10 monclient(hunting): my global_id is 4296 2024-03-29T10:46:56.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67b4004eb0 con 0x7f67c80ad850 2024-03-29T10:46:56.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 10 client.?.objecter ms_handle_connect 0x7f67c80b0770 2024-03-29T10:46:56.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:56.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974056817 0 0) 0x7f67c806fd30 con 0x7f67c813c1b0 2024-03-29T10:46:56.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:56.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:56.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:56.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 10 monclient(hunting): my global_id is 4315 2024-03-29T10:46:56.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.873+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c8135250 con 0x7f67c813c1b0 2024-03-29T10:46:56.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165574402 0 0) 0x7f67c81d19e0 con 0x7f67c80b0770 2024-03-29T10:46:56.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:56.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:56.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:56.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 10 monclient(hunting): my global_id is 4310 2024-03-29T10:46:56.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c806fd30 con 0x7f67c80b0770 2024-03-29T10:46:56.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1968451457 0 0) 0x7f67c8135250 con 0x7f67c813c1b0 2024-03-29T10:46:56.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67c81d19e0 con 0x7f67c813c1b0 2024-03-29T10:46:56.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f67c0001e60 con 0x7f67c813c1b0 2024-03-29T10:46:56.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:56.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:56.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 10 monclient(hunting): dump: 2024-03-29T10:46:56.865 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:56.865 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:56.865 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:56.866 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:56.866 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:56.866 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:56.866 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:56.866 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:56.866 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:56.866 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:56.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:56.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2711620717 0 0) 0x7f67c81d19e0 con 0x7f67c813c1b0 2024-03-29T10:46:56.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 >> v1:172.21.15.98:6790/0 conn(0x7f67c80b0770 legacy=0x7f67c8134430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:56.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.874+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 >> v1:172.21.15.98:6789/0 conn(0x7f67c80ad850 legacy=0x7f67c81d3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:56.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:46:56.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 1 monclient: found mon.b 2024-03-29T10:46:56.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:56.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:56.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c81d45b0 con 0x7f67c813c1b0 2024-03-29T10:46:56.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:46:56.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:46:56.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:56.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67ce6b3640 5 monclient: authenticate success, global_id 4315 2024-03-29T10:46:56.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67ce6b3640 10 monclient: _renew_subs 2024-03-29T10:46:56.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67ce6b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:56.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67ce6b3640 1 -- 172.21.15.98:0/4283351168 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c806f950 con 0x7f67c813c1b0 2024-03-29T10:46:56.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67ce6b3640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:56.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67ce6b3640 10 monclient: _renew_subs 2024-03-29T10:46:56.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67ce6b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:56.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67ce6b3640 1 -- 172.21.15.98:0/4283351168 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c806f670 con 0x7f67c813c1b0 2024-03-29T10:46:56.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67ce6b3640 1 librados: init done 2024-03-29T10:46:56.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67c0002060 con 0x7f67c813c1b0 2024-03-29T10:46:56.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:56.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c67fc640 4 set_mon_vals no callback set 2024-03-29T10:46:56.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f67c0001470 con 0x7f67c813c1b0 2024-03-29T10:46:56.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:56.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:56.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.875+0000 7f67c4ff9640 10 monclient: dump: 2024-03-29T10:46:56.869 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:46:56.869 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:46:56.869 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:56.869 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:46:56.869 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:56.869 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:46:56.869 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:46:56.869 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:46:56.869 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:46:56.869 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:46:56.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.876+0000 7f67ce6b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:56.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.876+0000 7f67ce6b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:56.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.877+0000 7f67ce6b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:56.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.877+0000 7f67ce6b3640 1 -- 172.21.15.98:0/4283351168 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a8005080 con 0x7f67c813c1b0 2024-03-29T10:46:56.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.877+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f67c0019cb0 con 0x7f67c813c1b0 2024-03-29T10:46:56.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.878+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f67c0083c50 con 0x7f67c813c1b0 2024-03-29T10:46:56.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.878+0000 7f67c4ff9640 10 client.4315.objecter ms_dispatch 0x7f67c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:56.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.878+0000 7f67c4ff9640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:56.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.878+0000 7f67c4ff9640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:56.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.878+0000 7f67c4ff9640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-29T10:46:56.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.879+0000 7f67c7fff640 10 monclient: get_auth_request con 0x7f67b406de50 auth_method 0 2024-03-29T10:46:56.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.879+0000 7f67c4ff9640 10 client.4315.objecter ms_handle_connect 0x7f67b406de50 2024-03-29T10:46:56.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.884+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67c004f360 con 0x7f67c813c1b0 2024-03-29T10:46:56.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.885+0000 7f67c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:56.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:56.885+0000 7f67c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:57.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.280+0000 7f67ce6b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:57.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.280+0000 7f67ce6b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:57.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.280+0000 7f67ce6b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:46:57.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.280+0000 7f67ce6b3640 1 -- 172.21.15.98:0/4283351168 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67a8005330 con 0x7f67c813c1b0 2024-03-29T10:46:57.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.281+0000 7f67c4ff9640 1 -- 172.21.15.98:0/4283351168 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f67c0051460 con 0x7f67c813c1b0 2024-03-29T10:46:57.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.282+0000 7f67c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:57.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.282+0000 7f67c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:57.264 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:46:57.264 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:57.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.285+0000 7f67ce6b3640 10 librados: watch_flush enter 2024-03-29T10:46:57.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.285+0000 7f67ce6b3640 10 librados: watch_flush exit 2024-03-29T10:46:57.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.285+0000 7f67ce6b3640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-29T10:46:57.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.285+0000 7f67ce6b3640 10 client.4315.objecter successfully canceled tick 2024-03-29T10:46:57.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.285+0000 7f67ce6b3640 1 -- 172.21.15.98:0/4283351168 >> v1:172.21.15.98:6816/1481799426 conn(0x7f67b406de50 legacy=0x7f67b4070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:57.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.285+0000 7f67ce6b3640 10 monclient: shutdown 2024-03-29T10:46:57.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.285+0000 7f67ce6b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:57.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.285+0000 7f67ce6b3640 1 -- 172.21.15.98:0/4283351168 >> v1:172.21.15.189:6789/0 conn(0x7f67c813c1b0 legacy=0x7f67c8134b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:57.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.285+0000 7f67ce6b3640 1 -- 172.21.15.98:0/4283351168 shutdown_connections 2024-03-29T10:46:57.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.285+0000 7f67ce6b3640 1 -- 172.21.15.98:0/4283351168 >> 172.21.15.98:0/4283351168 conn(0x7f67c806b660 msgr2=0x7f67c806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:57.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.285+0000 7f67ce6b3640 1 -- 172.21.15.98:0/4283351168 shutdown_connections 2024-03-29T10:46:57.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.286+0000 7f67ce6b3640 1 -- 172.21.15.98:0/4283351168 wait complete. 2024-03-29T10:46:57.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:46:57.286+0000 7f67ce6b3640 1 librados: shutdown 2024-03-29T10:46:57.282 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:04.089 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:04.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.258+0000 7ff9c1440640 10 monclient: get_monmap_and_config 2024-03-29T10:47:04.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.258+0000 7ff9c1440640 10 monclient: build_initial_monmap 2024-03-29T10:47:04.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.258+0000 7ff9c1440640 10 monclient: monmap: 2024-03-29T10:47:04.243 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:04.243 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:04.243 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:47:04.259052+0000 2024-03-29T10:47:04.243 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:47:04.259052+0000 2024-03-29T10:47:04.243 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:04.243 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:04.243 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:04.243 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:47:04.243 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:47:04.243 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:04.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.259+0000 7ff9c1440640 1 Processor -- start 2024-03-29T10:47:04.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.260+0000 7ff9c1440640 1 -- start start 2024-03-29T10:47:04.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.260+0000 7ff9c1440640 10 monclient: init 2024-03-29T10:47:04.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.260+0000 7ff9c1440640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:04.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.260+0000 7ff9c1440640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:04.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.260+0000 7ff9c1440640 10 monclient(hunting): picked mon.noname-b con 0x7ff9bc1457f0 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:04.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.260+0000 7ff9c1440640 10 monclient(hunting): picked mon.noname-c con 0x7ff9bc13f650 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:04.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.260+0000 7ff9c1440640 10 monclient(hunting): picked mon.noname-a con 0x7ff9bc0a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:04.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.260+0000 7ff9c1440640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9bc0aaef0 con 0x7ff9bc0a9d70 2024-03-29T10:47:04.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.261+0000 7ff9c1440640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9bc145bd0 con 0x7ff9bc13f650 2024-03-29T10:47:04.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.261+0000 7ff9c1440640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9bc145eb0 con 0x7ff9bc1457f0 2024-03-29T10:47:04.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.261+0000 7ff9c1440640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:04.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.261+0000 7ff9c1440640 10 monclient(hunting): authenticate will time out at 1044.458590s 2024-03-29T10:47:04.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.261+0000 7ff9bb7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff9bc0a9d70 0x7ff9bc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46928/0 (socket says 172.21.15.98:46928) 2024-03-29T10:47:04.245 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.261+0000 7ff9bb7fe640 1 -- 172.21.15.98:0/3705521708 learned_addr learned my addr 172.21.15.98:0/3705521708 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:04.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.262+0000 7ff9b9ffb640 1 -- 172.21.15.98:0/3705521708 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248512135 0 0) 0x7ff9bc0aaef0 con 0x7ff9bc0a9d70 2024-03-29T10:47:04.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.262+0000 7ff9b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:04.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.262+0000 7ff9b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:04.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.262+0000 7ff9b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:04.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.262+0000 7ff9b9ffb640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:47:04.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.262+0000 7ff9b9ffb640 1 -- 172.21.15.98:0/3705521708 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a4003520 con 0x7ff9bc0a9d70 2024-03-29T10:47:04.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.262+0000 7ff9b9ffb640 1 -- 172.21.15.98:0/3705521708 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4178568304 0 0) 0x7ff9a4003520 con 0x7ff9bc0a9d70 2024-03-29T10:47:04.246 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.262+0000 7ff9b9ffb640 1 -- 172.21.15.98:0/3705521708 >> v1:172.21.15.98:6790/0 conn(0x7ff9bc1457f0 legacy=0x7ff9bc13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:04.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.267+0000 7ff9b9ffb640 1 -- 172.21.15.98:0/3705521708 >> v1:172.21.15.189:6789/0 conn(0x7ff9bc13f650 legacy=0x7ff9bc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:04.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.267+0000 7ff9b9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:04.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.267+0000 7ff9b9ffb640 1 monclient: found mon.noname-a 2024-03-29T10:47:04.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.267+0000 7ff9b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:04.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:47:04.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 1 -- 172.21.15.98:0/3705521708 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9bc146340 con 0x7ff9bc0a9d70 2024-03-29T10:47:04.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:04.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:04.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:04.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 1 -- 172.21.15.98:0/3705521708 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff9b4001a10 con 0x7ff9bc0a9d70 2024-03-29T10:47:04.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:04.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:04.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 10 monclient: dump: 2024-03-29T10:47:04.251 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:04.251 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:04.251 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:04.251 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:04.251 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:04.251 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:04.251 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:04.251 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:04.251 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:04.251 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:04.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9c1440640 5 monclient: authenticate success, global_id 4299 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9c1440640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 1 -- 172.21.15.98:0/3705521708 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9b4001be0 con 0x7ff9bc0a9d70 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 1 -- 172.21.15.98:0/3705521708 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff9b4001ff0 con 0x7ff9bc0a9d70 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9b9ffb640 10 monclient: dump: 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:04.253 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:04.254 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:04.254 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:04.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9c1440640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:04.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.268+0000 7ff9c1440640 4 set_mon_vals no callback set 2024-03-29T10:47:04.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.269+0000 7ff9c1440640 10 monclient: shutdown 2024-03-29T10:47:04.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.269+0000 7ff9c1440640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:04.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.269+0000 7ff9c1440640 1 -- 172.21.15.98:0/3705521708 >> v1:172.21.15.98:6789/0 conn(0x7ff9bc0a9d70 legacy=0x7ff9bc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:04.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.269+0000 7ff9c1440640 1 -- 172.21.15.98:0/3705521708 shutdown_connections 2024-03-29T10:47:04.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.269+0000 7ff9c1440640 1 -- 172.21.15.98:0/3705521708 >> 172.21.15.98:0/3705521708 conn(0x7ff9bc066a00 msgr2=0x7ff9bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:04.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.269+0000 7ff9c1440640 1 -- 172.21.15.98:0/3705521708 shutdown_connections 2024-03-29T10:47:04.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.269+0000 7ff9c1440640 1 -- 172.21.15.98:0/3705521708 wait complete. 2024-03-29T10:47:04.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.270+0000 7ff9c1440640 10 monclient: build_initial_monmap 2024-03-29T10:47:04.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.270+0000 7ff9c1440640 10 monclient: monmap: 2024-03-29T10:47:04.256 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:04.256 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:04.256 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:47:04.256 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:47:04.256 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:04.256 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:04.256 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:04.256 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:47:04.256 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:47:04.256 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:04.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.271+0000 7ff9c1440640 1 librados: starting msgr at 2024-03-29T10:47:04.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.271+0000 7ff9c1440640 1 librados: starting objecter 2024-03-29T10:47:04.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.271+0000 7ff9c1440640 1 Processor -- start 2024-03-29T10:47:04.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.273+0000 7ff9c1440640 1 -- start start 2024-03-29T10:47:04.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.273+0000 7ff9c1440640 1 librados: setting wanted keys 2024-03-29T10:47:04.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.273+0000 7ff9c1440640 1 librados: calling monclient init 2024-03-29T10:47:04.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.273+0000 7ff9c1440640 10 monclient: init 2024-03-29T10:47:04.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.273+0000 7ff9c1440640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:04.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.273+0000 7ff9c1440640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:04.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.273+0000 7ff9c1440640 10 monclient(hunting): picked mon.noname-b con 0x7ff9bc0a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:04.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.275+0000 7ff9c1440640 10 monclient(hunting): picked mon.noname-c con 0x7ff9bc13f650 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:04.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.275+0000 7ff9c1440640 10 monclient(hunting): picked mon.noname-a con 0x7ff9bc1457f0 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:04.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.275+0000 7ff9c1440640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9bc13b620 con 0x7ff9bc1457f0 2024-03-29T10:47:04.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.275+0000 7ff9bb7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff9bc1457f0 0x7ff9bc1e6200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47256/0 (socket says 172.21.15.98:47256) 2024-03-29T10:47:04.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.275+0000 7ff9bb7fe640 1 -- 172.21.15.98:0/3880962237 learned_addr learned my addr 172.21.15.98:0/3880962237 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:04.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.275+0000 7ff9c1440640 1 -- 172.21.15.98:0/3880962237 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9bc142a20 con 0x7ff9bc0a9d70 2024-03-29T10:47:04.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.276+0000 7ff9a37fe640 10 client.?.objecter ms_handle_connect 0x7ff9bc1457f0 2024-03-29T10:47:04.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.276+0000 7ff9a37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:04.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.276+0000 7ff9c1440640 1 -- 172.21.15.98:0/3880962237 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9bc142bb0 con 0x7ff9bc13f650 2024-03-29T10:47:04.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.276+0000 7ff9c1440640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:04.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.276+0000 7ff9c1440640 10 monclient(hunting): authenticate will time out at 1044.473054s 2024-03-29T10:47:04.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.276+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338771563 0 0) 0x7ff9bc13b620 con 0x7ff9bc1457f0 2024-03-29T10:47:04.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.276+0000 7ff9a37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:04.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.276+0000 7ff9a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:04.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.276+0000 7ff9a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:04.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.276+0000 7ff9a37fe640 10 monclient(hunting): my global_id is 4302 2024-03-29T10:47:04.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.276+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a8004eb0 con 0x7ff9bc1457f0 2024-03-29T10:47:04.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.278+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 879436948 0 0) 0x7ff9a8004eb0 con 0x7ff9bc1457f0 2024-03-29T10:47:04.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.278+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9bc13b620 con 0x7ff9bc1457f0 2024-03-29T10:47:04.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.278+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff9b4001ef0 con 0x7ff9bc1457f0 2024-03-29T10:47:04.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.278+0000 7ff9a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:04.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.278+0000 7ff9a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:04.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.278+0000 7ff9a37fe640 10 monclient(hunting): dump: 2024-03-29T10:47:04.262 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:04.262 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:04.262 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:04.262 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.278+0000 7ff9a37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3665652701 0 0) 0x7ff9bc13b620 con 0x7ff9bc1457f0 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 >> v1:172.21.15.98:6790/0 conn(0x7ff9bc13f650 legacy=0x7ff9bc141ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 >> v1:172.21.15.189:6789/0 conn(0x7ff9bc0a9d70 legacy=0x7ff9bc1db7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 1 monclient: found mon.a 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:04.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9bc142d40 con 0x7ff9bc1457f0 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9c1440640 5 monclient: authenticate success, global_id 4302 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9c1440640 10 monclient: _renew_subs 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9c1440640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9c1440640 1 -- 172.21.15.98:0/3880962237 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9bc13b000 con 0x7ff9bc1457f0 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9b4004f70 con 0x7ff9bc1457f0 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff9b4005280 con 0x7ff9bc1457f0 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9a37fe640 10 monclient: dump: 2024-03-29T10:47:04.264 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9c1440640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9c1440640 10 monclient: _renew_subs 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9c1440640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9c1440640 1 -- 172.21.15.98:0/3880962237 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9bc06de90 con 0x7ff9bc1457f0 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9c1440640 1 librados: init done 2024-03-29T10:47:04.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.279+0000 7ff9b8ff9640 4 set_mon_vals no callback set 2024-03-29T10:47:04.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.280+0000 7ff9c1440640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:04.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.280+0000 7ff9c1440640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:04.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.280+0000 7ff9c1440640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:04.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.280+0000 7ff9c1440640 1 -- 172.21.15.98:0/3880962237 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99c005080 con 0x7ff9bc1457f0 2024-03-29T10:47:04.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.283+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7ff9b401b250 con 0x7ff9bc1457f0 2024-03-29T10:47:04.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.284+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7ff9b4085010 con 0x7ff9bc1457f0 2024-03-29T10:47:04.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.284+0000 7ff9a37fe640 10 client.4302.objecter ms_dispatch 0x7ff9bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:04.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.284+0000 7ff9a37fe640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:04.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.284+0000 7ff9a37fe640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:04.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.284+0000 7ff9a37fe640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-29T10:47:04.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.284+0000 7ff9baffd640 10 monclient: get_auth_request con 0x7ff9a806db90 auth_method 0 2024-03-29T10:47:04.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.285+0000 7ff9a37fe640 10 client.4302.objecter ms_handle_connect 0x7ff9a806db90 2024-03-29T10:47:04.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.294+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff9b40508f0 con 0x7ff9bc1457f0 2024-03-29T10:47:04.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.294+0000 7ff9a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:04.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.294+0000 7ff9a37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:04.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.620+0000 7ff9c1440640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:04.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.620+0000 7ff9c1440640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:04.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.620+0000 7ff9c1440640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:04.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.620+0000 7ff9c1440640 1 -- 172.21.15.98:0/3880962237 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff99c005330 con 0x7ff9bc1457f0 2024-03-29T10:47:04.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.623+0000 7ff9a37fe640 1 -- 172.21.15.98:0/3880962237 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7ff9b4052820 con 0x7ff9bc1457f0 2024-03-29T10:47:04.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.623+0000 7ff9a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:04.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.623+0000 7ff9a37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:04.606 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:47:04.606 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:04.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.627+0000 7ff9c1440640 10 librados: watch_flush enter 2024-03-29T10:47:04.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.627+0000 7ff9c1440640 10 librados: watch_flush exit 2024-03-29T10:47:04.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.627+0000 7ff9c1440640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-29T10:47:04.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.627+0000 7ff9c1440640 10 client.4302.objecter successfully canceled tick 2024-03-29T10:47:04.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.628+0000 7ff9c1440640 1 -- 172.21.15.98:0/3880962237 >> v1:172.21.15.98:6816/1481799426 conn(0x7ff9a806db90 legacy=0x7ff9a8070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:04.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.628+0000 7ff9c1440640 10 monclient: shutdown 2024-03-29T10:47:04.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.628+0000 7ff9c1440640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:04.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.628+0000 7ff9c1440640 1 -- 172.21.15.98:0/3880962237 >> v1:172.21.15.98:6789/0 conn(0x7ff9bc1457f0 legacy=0x7ff9bc1e6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:04.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.628+0000 7ff9c1440640 1 -- 172.21.15.98:0/3880962237 shutdown_connections 2024-03-29T10:47:04.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.628+0000 7ff9c1440640 1 -- 172.21.15.98:0/3880962237 >> 172.21.15.98:0/3880962237 conn(0x7ff9bc066a00 msgr2=0x7ff9bc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:04.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.628+0000 7ff9c1440640 1 -- 172.21.15.98:0/3880962237 shutdown_connections 2024-03-29T10:47:04.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.629+0000 7ff9c1440640 1 -- 172.21.15.98:0/3880962237 wait complete. 2024-03-29T10:47:04.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:04.629+0000 7ff9c1440640 1 librados: shutdown 2024-03-29T10:47:04.629 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:11.436 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:11.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.602+0000 7f6041faf640 10 monclient: get_monmap_and_config 2024-03-29T10:47:11.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.602+0000 7f6041faf640 10 monclient: build_initial_monmap 2024-03-29T10:47:11.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.602+0000 7f6041faf640 10 monclient: monmap: 2024-03-29T10:47:11.585 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:11.585 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:11.585 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:47:11.603052+0000 2024-03-29T10:47:11.585 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:47:11.603052+0000 2024-03-29T10:47:11.586 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:11.586 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:11.586 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:11.586 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:47:11.586 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:47:11.586 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:11.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.603+0000 7f6041faf640 1 Processor -- start 2024-03-29T10:47:11.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.603+0000 7f6041faf640 1 -- start start 2024-03-29T10:47:11.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.603+0000 7f6041faf640 10 monclient: init 2024-03-29T10:47:11.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.604+0000 7f6041faf640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:11.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.604+0000 7f6041faf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:11.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.604+0000 7f6041faf640 10 monclient(hunting): picked mon.noname-b con 0x7f603c1422b0 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:11.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.604+0000 7f6041faf640 10 monclient(hunting): picked mon.noname-c con 0x7f603c142690 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:11.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.604+0000 7f6041faf640 10 monclient(hunting): picked mon.noname-a con 0x7f603c0b17d0 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:11.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.604+0000 7f6041faf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c142c80 con 0x7f603c0b17d0 2024-03-29T10:47:11.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.604+0000 7f6041faf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c0aae40 con 0x7f603c142690 2024-03-29T10:47:11.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.604+0000 7f6041faf640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c13fbd0 con 0x7f603c1422b0 2024-03-29T10:47:11.588 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.604+0000 7f6041faf640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:11.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.604+0000 7f6041faf640 10 monclient(hunting): authenticate will time out at 1051.801464s 2024-03-29T10:47:11.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.606+0000 7f603bfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f603c0b17d0 0x7f603c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41927/0 (socket says 172.21.15.98:41927) 2024-03-29T10:47:11.589 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.607+0000 7f603bfff640 1 -- 172.21.15.98:0/289683753 learned_addr learned my addr 172.21.15.98:0/289683753 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:11.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 1 -- 172.21.15.98:0/289683753 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159103385 0 0) 0x7f603c142c80 con 0x7f603c0b17d0 2024-03-29T10:47:11.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:11.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:11.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:11.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 10 monclient(hunting): my global_id is 4305 2024-03-29T10:47:11.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 1 -- 172.21.15.98:0/289683753 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6024003520 con 0x7f603c0b17d0 2024-03-29T10:47:11.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 1 -- 172.21.15.98:0/289683753 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3589373738 0 0) 0x7f6024003520 con 0x7f603c0b17d0 2024-03-29T10:47:11.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 1 -- 172.21.15.98:0/289683753 >> v1:172.21.15.98:6790/0 conn(0x7f603c1422b0 legacy=0x7f603c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:11.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 1 -- 172.21.15.98:0/289683753 >> v1:172.21.15.189:6789/0 conn(0x7f603c142690 legacy=0x7f603c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:11.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:11.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 1 monclient: found mon.noname-a 2024-03-29T10:47:11.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:11.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:47:11.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 1 -- 172.21.15.98:0/289683753 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c151940 con 0x7f603c0b17d0 2024-03-29T10:47:11.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:11.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:11.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:11.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 1 -- 172.21.15.98:0/289683753 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f602c001690 con 0x7f603c0b17d0 2024-03-29T10:47:11.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:11.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:11.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.609+0000 7f603a7fc640 10 monclient: dump: 2024-03-29T10:47:11.594 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:11.594 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:11.594 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:11.594 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:11.594 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.610+0000 7f6041faf640 5 monclient: authenticate success, global_id 4305 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.610+0000 7f6041faf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.610+0000 7f603a7fc640 1 -- 172.21.15.98:0/289683753 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f602c0017f0 con 0x7f603c0b17d0 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.610+0000 7f603a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.610+0000 7f603a7fc640 1 -- 172.21.15.98:0/289683753 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f602c001c40 con 0x7f603c0b17d0 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.610+0000 7f603a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.610+0000 7f603a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.610+0000 7f603a7fc640 10 monclient: dump: 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:11.595 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:11.596 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:11.596 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:11.596 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:11.596 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:11.596 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:11.596 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:11.596 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:11.596 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:11.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.610+0000 7f6041faf640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:11.596 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.610+0000 7f6041faf640 4 set_mon_vals no callback set 2024-03-29T10:47:11.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.611+0000 7f6041faf640 10 monclient: shutdown 2024-03-29T10:47:11.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.611+0000 7f6041faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.611+0000 7f6041faf640 1 -- 172.21.15.98:0/289683753 >> v1:172.21.15.98:6789/0 conn(0x7f603c0b17d0 legacy=0x7f603c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.611+0000 7f6041faf640 1 -- 172.21.15.98:0/289683753 shutdown_connections 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.611+0000 7f6041faf640 1 -- 172.21.15.98:0/289683753 >> 172.21.15.98:0/289683753 conn(0x7f603c066a00 msgr2=0x7f603c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.611+0000 7f6041faf640 1 -- 172.21.15.98:0/289683753 shutdown_connections 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.611+0000 7f6041faf640 1 -- 172.21.15.98:0/289683753 wait complete. 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.612+0000 7f6041faf640 10 monclient: build_initial_monmap 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.612+0000 7f6041faf640 10 monclient: monmap: 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:11.600 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.612+0000 7f6041faf640 1 librados: starting msgr at 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.612+0000 7f6041faf640 1 librados: starting objecter 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.612+0000 7f6041faf640 1 Processor -- start 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.613+0000 7f6041faf640 1 -- start start 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.613+0000 7f6041faf640 1 librados: setting wanted keys 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.613+0000 7f6041faf640 1 librados: calling monclient init 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.613+0000 7f6041faf640 10 monclient: init 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.614+0000 7f6041faf640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.614+0000 7f6041faf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.614+0000 7f6041faf640 10 monclient(hunting): picked mon.noname-c con 0x7f603c0b17d0 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:11.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.614+0000 7f6041faf640 10 monclient(hunting): picked mon.noname-b con 0x7f603c1422b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:11.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.614+0000 7f6041faf640 10 monclient(hunting): picked mon.noname-a con 0x7f603c142690 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:11.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.614+0000 7f6041faf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c06e860 con 0x7f603c142690 2024-03-29T10:47:11.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.614+0000 7f6041faf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c150460 con 0x7f603c1422b0 2024-03-29T10:47:11.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.614+0000 7f6041faf640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c06eab0 con 0x7f603c0b17d0 2024-03-29T10:47:11.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.614+0000 7f6041faf640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:11.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.614+0000 7f6041faf640 10 monclient(hunting): authenticate will time out at 1051.811296s 2024-03-29T10:47:11.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.614+0000 7f603b7fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f603c0b17d0 0x7f603c1508f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:33004/0 (socket says 172.21.15.98:33004) 2024-03-29T10:47:11.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.614+0000 7f603b7fe640 1 -- 172.21.15.98:0/2899656689 learned_addr learned my addr 172.21.15.98:0/2899656689 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:11.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.617+0000 7f6023fff640 10 client.?.objecter ms_handle_connect 0x7f603c0b17d0 2024-03-29T10:47:11.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.617+0000 7f6023fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:11.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.618+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755897026 0 0) 0x7f603c06eab0 con 0x7f603c0b17d0 2024-03-29T10:47:11.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.618+0000 7f6023fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:11.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.618+0000 7f6023fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.618+0000 7f6023fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.618+0000 7f6023fff640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.618+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6028004eb0 con 0x7f603c0b17d0 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 10 client.?.objecter ms_handle_connect 0x7f603c1422b0 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231669789 0 0) 0x7f603c150460 con 0x7f603c1422b0 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 10 monclient(hunting): my global_id is 4321 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f603c06eab0 con 0x7f603c1422b0 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3402720555 0 0) 0x7f603c06eab0 con 0x7f603c1422b0 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f603c150460 con 0x7f603c1422b0 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f6034001b10 con 0x7f603c1422b0 2024-03-29T10:47:11.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 10 monclient(hunting): dump: 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:11.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1438767331 0 0) 0x7f603c150460 con 0x7f603c1422b0 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 >> v1:172.21.15.98:6790/0 conn(0x7f603c0b17d0 legacy=0x7f603c1508f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 >> v1:172.21.15.98:6789/0 conn(0x7f603c142690 legacy=0x7f603c1e6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 1 monclient: found mon.b 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.619+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c06ed90 con 0x7f603c1422b0 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6023fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6023fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6023fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6041faf640 5 monclient: authenticate success, global_id 4321 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6041faf640 10 monclient: _renew_subs 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6041faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:47:11.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6041faf640 1 -- 172.21.15.98:0/2899656689 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603c1de9d0 con 0x7f603c1422b0 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6041faf640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6041faf640 10 monclient: _renew_subs 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6041faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6041faf640 1 -- 172.21.15.98:0/2899656689 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f603c1511e0 con 0x7f603c1422b0 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6041faf640 1 librados: init done 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60340020e0 con 0x7f603c1422b0 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6023fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f60397fa640 4 set_mon_vals no callback set 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f6034003b10 con 0x7f603c1422b0 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6023fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6023fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.620+0000 7f6023fff640 10 monclient: dump: 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:11.611 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.621+0000 7f6041faf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.621+0000 7f6041faf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.621+0000 7f6041faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.621+0000 7f6041faf640 1 -- 172.21.15.98:0/2899656689 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f601c005080 con 0x7f603c1422b0 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.622+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f6034003c70 con 0x7f603c1422b0 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.623+0000 7f603b7fe640 10 monclient: get_auth_request con 0x7f602806ddc0 auth_method 0 2024-03-29T10:47:11.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.624+0000 7f6023fff640 10 client.4321.objecter ms_handle_connect 0x7f602806ddc0 2024-03-29T10:47:11.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.624+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f6034083b10 con 0x7f603c1422b0 2024-03-29T10:47:11.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.624+0000 7f6023fff640 10 client.4321.objecter ms_dispatch 0x7f603c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:11.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.624+0000 7f6023fff640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:11.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.624+0000 7f6023fff640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:11.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.625+0000 7f6023fff640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-29T10:47:11.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.631+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f603404f3f0 con 0x7f603c1422b0 2024-03-29T10:47:11.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.632+0000 7f6023fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:11.614 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.632+0000 7f6023fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:11.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.962+0000 7f6041faf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:11.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.962+0000 7f6041faf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:11.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.963+0000 7f6041faf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:47:11.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.963+0000 7f6041faf640 1 -- 172.21.15.98:0/2899656689 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f601c005330 con 0x7f603c1422b0 2024-03-29T10:47:11.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.964+0000 7f6023fff640 1 -- 172.21.15.98:0/2899656689 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f6034051320 con 0x7f603c1422b0 2024-03-29T10:47:11.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.964+0000 7f6023fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:11.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.964+0000 7f6023fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:11.946 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:47:11.946 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:11.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.967+0000 7f6041faf640 10 librados: watch_flush enter 2024-03-29T10:47:11.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.967+0000 7f6041faf640 10 librados: watch_flush exit 2024-03-29T10:47:11.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.967+0000 7f6041faf640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-29T10:47:11.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.967+0000 7f6041faf640 10 client.4321.objecter successfully canceled tick 2024-03-29T10:47:11.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.967+0000 7f6041faf640 1 -- 172.21.15.98:0/2899656689 >> v1:172.21.15.98:6816/1481799426 conn(0x7f602806ddc0 legacy=0x7f6028070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:11.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.967+0000 7f6041faf640 10 monclient: shutdown 2024-03-29T10:47:11.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.967+0000 7f6041faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:11.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.967+0000 7f6041faf640 1 -- 172.21.15.98:0/2899656689 >> v1:172.21.15.189:6789/0 conn(0x7f603c1422b0 legacy=0x7f603c071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:11.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.967+0000 7f6041faf640 1 -- 172.21.15.98:0/2899656689 shutdown_connections 2024-03-29T10:47:11.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.967+0000 7f6041faf640 1 -- 172.21.15.98:0/2899656689 >> 172.21.15.98:0/2899656689 conn(0x7f603c066a00 msgr2=0x7f603c13c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:11.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.968+0000 7f6041faf640 1 -- 172.21.15.98:0/2899656689 shutdown_connections 2024-03-29T10:47:11.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.968+0000 7f6041faf640 1 -- 172.21.15.98:0/2899656689 wait complete. 2024-03-29T10:47:11.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:11.968+0000 7f6041faf640 1 librados: shutdown 2024-03-29T10:47:11.967 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:18.774 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:18.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.943+0000 7fee1359e640 10 monclient: get_monmap_and_config 2024-03-29T10:47:18.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.943+0000 7fee1359e640 10 monclient: build_initial_monmap 2024-03-29T10:47:18.926 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.943+0000 7fee1359e640 10 monclient: monmap: 2024-03-29T10:47:18.926 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:18.926 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:18.926 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:47:18.944527+0000 2024-03-29T10:47:18.926 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:47:18.944527+0000 2024-03-29T10:47:18.926 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:18.926 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:18.926 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:18.926 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:47:18.927 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:47:18.927 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:18.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.944+0000 7fee1359e640 1 Processor -- start 2024-03-29T10:47:18.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.944+0000 7fee1359e640 1 -- start start 2024-03-29T10:47:18.927 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.944+0000 7fee1359e640 10 monclient: init 2024-03-29T10:47:18.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.945+0000 7fee1359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:18.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.945+0000 7fee1359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:18.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.945+0000 7fee1359e640 10 monclient(hunting): picked mon.noname-c con 0x7fee0c0389e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:18.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.945+0000 7fee1359e640 10 monclient(hunting): picked mon.noname-a con 0x7fee0c038dc0 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:18.928 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.946+0000 7fee1359e640 10 monclient(hunting): picked mon.noname-b con 0x7fee0c030660 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:18.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.946+0000 7fee1359e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0c54c0 con 0x7fee0c038dc0 2024-03-29T10:47:18.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.946+0000 7fee1359e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c036430 con 0x7fee0c0389e0 2024-03-29T10:47:18.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.946+0000 7fee1359e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c039840 con 0x7fee0c030660 2024-03-29T10:47:18.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.946+0000 7fee1359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:18.929 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.946+0000 7fee1359e640 10 monclient(hunting): authenticate will time out at 1059.143013s 2024-03-29T10:47:18.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.947+0000 7fee12d9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fee0c030660 0x7fee0c0d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:52180/0 (socket says 172.21.15.98:52180) 2024-03-29T10:47:18.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.947+0000 7fee12d9d640 1 -- 172.21.15.98:0/1717386359 learned_addr learned my addr 172.21.15.98:0/1717386359 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:18.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.948+0000 7fee1159a640 1 -- 172.21.15.98:0/1717386359 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380086825 0 0) 0x7fee0c039840 con 0x7fee0c030660 2024-03-29T10:47:18.930 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.948+0000 7fee1159a640 10 monclient(hunting): _negotiate 2024-03-29T10:47:18.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.948+0000 7fee1159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:18.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.948+0000 7fee1159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:18.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.948+0000 7fee1159a640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:47:18.931 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.948+0000 7fee1159a640 1 -- 172.21.15.98:0/1717386359 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedfc003520 con 0x7fee0c030660 2024-03-29T10:47:18.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.948+0000 7fee1159a640 1 -- 172.21.15.98:0/1717386359 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3769230117 0 0) 0x7fedfc003520 con 0x7fee0c030660 2024-03-29T10:47:18.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.948+0000 7fee1159a640 1 -- 172.21.15.98:0/1717386359 >> v1:172.21.15.189:6789/0 conn(0x7fee0c0389e0 legacy=0x7fee0c033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:18.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1159a640 1 -- 172.21.15.98:0/1717386359 >> v1:172.21.15.98:6789/0 conn(0x7fee0c038dc0 legacy=0x7fee0c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:18.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1159a640 10 monclient: _finish_hunting 0 2024-03-29T10:47:18.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1159a640 1 monclient: found mon.noname-b 2024-03-29T10:47:18.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:18.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:47:18.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1159a640 1 -- 172.21.15.98:0/1717386359 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c035690 con 0x7fee0c030660 2024-03-29T10:47:18.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1159a640 10 monclient: _finish_auth 0 2024-03-29T10:47:18.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1159a640 10 monclient: _check_auth_tickets 2024-03-29T10:47:18.932 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:18.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1159a640 1 -- 172.21.15.98:0/1717386359 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fee1404f730 con 0x7fee0c030660 2024-03-29T10:47:18.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:18.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:18.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1159a640 10 monclient: dump: 2024-03-29T10:47:18.933 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:18.933 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:18.933 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:18.933 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:18.933 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:18.933 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:18.933 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:18.933 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:18.933 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:18.934 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:18.934 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1359e640 5 monclient: authenticate success, global_id 4316 2024-03-29T10:47:18.934 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.949+0000 7fee1359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:18.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.951+0000 7fee1159a640 1 -- 172.21.15.98:0/1717386359 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee140008d0 con 0x7fee0c030660 2024-03-29T10:47:18.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.951+0000 7fee1159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:18.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.951+0000 7fee1159a640 1 -- 172.21.15.98:0/1717386359 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fee140758b0 con 0x7fee0c030660 2024-03-29T10:47:18.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.951+0000 7fee1159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:18.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.951+0000 7fee1159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:18.935 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.951+0000 7fee1159a640 10 monclient: dump: 2024-03-29T10:47:18.935 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:18.935 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:18.936 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:18.936 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:18.936 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:18.936 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:18.936 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:18.936 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:18.936 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:18.936 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:18.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.952+0000 7fee1359e640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:18.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.952+0000 7fee1359e640 4 set_mon_vals no callback set 2024-03-29T10:47:18.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.953+0000 7fee1359e640 10 monclient: shutdown 2024-03-29T10:47:18.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.953+0000 7fee1359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:18.936 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.953+0000 7fee1359e640 1 -- 172.21.15.98:0/1717386359 >> v1:172.21.15.98:6790/0 conn(0x7fee0c030660 legacy=0x7fee0c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:18.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.956+0000 7fee1359e640 1 -- 172.21.15.98:0/1717386359 shutdown_connections 2024-03-29T10:47:18.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.956+0000 7fee1359e640 1 -- 172.21.15.98:0/1717386359 >> 172.21.15.98:0/1717386359 conn(0x7fee0c005c00 msgr2=0x7fee0c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:18.938 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.956+0000 7fee1359e640 1 -- 172.21.15.98:0/1717386359 shutdown_connections 2024-03-29T10:47:18.939 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.956+0000 7fee1359e640 1 -- 172.21.15.98:0/1717386359 wait complete. 2024-03-29T10:47:18.940 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.957+0000 7fee1359e640 10 monclient: build_initial_monmap 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.957+0000 7fee1359e640 10 monclient: monmap: 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.958+0000 7fee1359e640 1 librados: starting msgr at 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.958+0000 7fee1359e640 1 librados: starting objecter 2024-03-29T10:47:18.941 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.958+0000 7fee1359e640 1 Processor -- start 2024-03-29T10:47:18.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.958+0000 7fee1359e640 1 -- start start 2024-03-29T10:47:18.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.958+0000 7fee1359e640 1 librados: setting wanted keys 2024-03-29T10:47:18.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.958+0000 7fee1359e640 1 librados: calling monclient init 2024-03-29T10:47:18.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.958+0000 7fee1359e640 10 monclient: init 2024-03-29T10:47:18.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.959+0000 7fee1359e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:18.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.959+0000 7fee1359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:18.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.959+0000 7fee1359e640 10 monclient(hunting): picked mon.noname-c con 0x7fee0c030660 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:18.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.959+0000 7fee1359e640 10 monclient(hunting): picked mon.noname-a con 0x7fee0c0389e0 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:18.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.959+0000 7fee1359e640 10 monclient(hunting): picked mon.noname-b con 0x7fee0c02f3c0 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:18.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.959+0000 7fee1359e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c009770 con 0x7fee0c0389e0 2024-03-29T10:47:18.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.959+0000 7fee1359e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0c3510 con 0x7fee0c02f3c0 2024-03-29T10:47:18.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.959+0000 7fee1359e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0c36a0 con 0x7fee0c030660 2024-03-29T10:47:18.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.959+0000 7fee1359e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:18.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.959+0000 7fee1359e640 10 monclient(hunting): authenticate will time out at 1059.156586s 2024-03-29T10:47:18.945 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.962+0000 7fee1259c640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fee0c030660 0x7fee0c164d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:54798/0 (socket says 172.21.15.98:54798) 2024-03-29T10:47:18.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.962+0000 7fee1259c640 1 -- 172.21.15.98:0/3466767386 learned_addr learned my addr 172.21.15.98:0/3466767386 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:18.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.962+0000 7fee02ffd640 10 client.?.objecter ms_handle_connect 0x7fee0c030660 2024-03-29T10:47:18.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.962+0000 7fee02ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:18.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.962+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478461776 0 0) 0x7fee0c0c36a0 con 0x7fee0c030660 2024-03-29T10:47:18.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.963+0000 7fee02ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:18.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.963+0000 7fee02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:18.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.963+0000 7fee02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.963+0000 7fee02ffd640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.963+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf4004eb0 con 0x7fee0c030660 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.964+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1301250803 0 0) 0x7fedf4004eb0 con 0x7fee0c030660 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.964+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee0c0c36a0 con 0x7fee0c030660 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.964+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fee08001870 con 0x7fee0c030660 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.964+0000 7fee02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.964+0000 7fee02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.964+0000 7fee02ffd640 10 monclient(hunting): dump: 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:18.949 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.964+0000 7fee02ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.964+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4002374052 0 0) 0x7fee0c0c36a0 con 0x7fee0c030660 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.964+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 >> v1:172.21.15.189:6789/0 conn(0x7fee0c02f3c0 legacy=0x7fee0c16c7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.964+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 >> v1:172.21.15.98:6789/0 conn(0x7fee0c0389e0 legacy=0x7fee0c0c2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 1 monclient: found mon.c 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c0c3830 con 0x7fee0c030660 2024-03-29T10:47:18.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 10 monclient: _finish_auth 0 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 10 client.?.objecter ms_handle_connect 0x7fee0c0389e0 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee080019d0 con 0x7fee0c030660 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fee08004ae0 con 0x7fee0c030660 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.965+0000 7fee02ffd640 10 monclient: dump: 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:18.951 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:18.952 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:18.952 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:18.952 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:18.952 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:18.952 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:18.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.966+0000 7fee10d99640 4 set_mon_vals no callback set 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.967+0000 7fee1359e640 5 monclient: authenticate success, global_id 4319 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.967+0000 7fee1359e640 10 monclient: _renew_subs 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.967+0000 7fee1359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.967+0000 7fee1359e640 1 -- 172.21.15.98:0/3466767386 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0c02fdf0 con 0x7fee0c030660 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.967+0000 7fee1359e640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.967+0000 7fee1359e640 10 monclient: _renew_subs 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.967+0000 7fee1359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.967+0000 7fee1359e640 1 -- 172.21.15.98:0/3466767386 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0c164900 con 0x7fee0c030660 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.967+0000 7fee1359e640 1 librados: init done 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.970+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fee0801aab0 con 0x7fee0c030660 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.970+0000 7fee1359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.970+0000 7fee1359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.970+0000 7fee1359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:47:18.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.970+0000 7fee1359e640 1 -- 172.21.15.98:0/3466767386 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0c007b90 con 0x7fee0c030660 2024-03-29T10:47:18.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.970+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fee08084b40 con 0x7fee0c030660 2024-03-29T10:47:18.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.970+0000 7fee02ffd640 10 client.4319.objecter ms_dispatch 0x7fee14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:18.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.970+0000 7fee02ffd640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:18.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.970+0000 7fee02ffd640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:18.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.971+0000 7fee02ffd640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-29T10:47:18.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.971+0000 7fee11d9b640 10 monclient: get_auth_request con 0x7fedf406db80 auth_method 0 2024-03-29T10:47:18.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.971+0000 7fee02ffd640 10 client.4319.objecter ms_handle_connect 0x7fedf406db80 2024-03-29T10:47:18.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.981+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee08050420 con 0x7fee0c030660 2024-03-29T10:47:18.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.981+0000 7fee02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:18.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:18.981+0000 7fee02ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:19.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.313+0000 7fee1359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:19.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.313+0000 7fee1359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:19.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.313+0000 7fee1359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:47:19.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.313+0000 7fee1359e640 1 -- 172.21.15.98:0/3466767386 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee0c030190 con 0x7fee0c030660 2024-03-29T10:47:19.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.317+0000 7fee02ffd640 1 -- 172.21.15.98:0/3466767386 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fee08052350 con 0x7fee0c030660 2024-03-29T10:47:19.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.317+0000 7fee02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:19.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.317+0000 7fee02ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:19.299 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:47:19.299 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:19.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.321+0000 7fee1359e640 10 librados: watch_flush enter 2024-03-29T10:47:19.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.321+0000 7fee1359e640 10 librados: watch_flush exit 2024-03-29T10:47:19.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.321+0000 7fee1359e640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-29T10:47:19.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.321+0000 7fee1359e640 10 client.4319.objecter successfully canceled tick 2024-03-29T10:47:19.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.321+0000 7fee1359e640 1 -- 172.21.15.98:0/3466767386 >> v1:172.21.15.98:6816/1481799426 conn(0x7fedf406db80 legacy=0x7fedf4070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:19.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.321+0000 7fee1359e640 10 monclient: shutdown 2024-03-29T10:47:19.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.321+0000 7fee1359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:19.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.321+0000 7fee1359e640 1 -- 172.21.15.98:0/3466767386 >> v1:172.21.15.98:6790/0 conn(0x7fee0c030660 legacy=0x7fee0c164d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:19.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.322+0000 7fee1359e640 1 -- 172.21.15.98:0/3466767386 shutdown_connections 2024-03-29T10:47:19.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.322+0000 7fee1359e640 1 -- 172.21.15.98:0/3466767386 >> 172.21.15.98:0/3466767386 conn(0x7fee0c005c00 msgr2=0x7fee0c034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:19.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.322+0000 7fee1359e640 1 -- 172.21.15.98:0/3466767386 shutdown_connections 2024-03-29T10:47:19.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.322+0000 7fee1359e640 1 -- 172.21.15.98:0/3466767386 wait complete. 2024-03-29T10:47:19.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:19.322+0000 7fee1359e640 1 librados: shutdown 2024-03-29T10:47:19.325 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:26.132 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:26.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.291+0000 7f3644214640 10 monclient: get_monmap_and_config 2024-03-29T10:47:26.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.291+0000 7f3644214640 10 monclient: build_initial_monmap 2024-03-29T10:47:26.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.291+0000 7f3644214640 10 monclient: monmap: 2024-03-29T10:47:26.283 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:26.283 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:26.283 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:47:26.293718+0000 2024-03-29T10:47:26.284 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:47:26.293718+0000 2024-03-29T10:47:26.284 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:26.284 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:26.284 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:26.284 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:47:26.284 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:47:26.285 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:26.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.292+0000 7f3644214640 1 Processor -- start 2024-03-29T10:47:26.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.292+0000 7f3644214640 1 -- start start 2024-03-29T10:47:26.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.292+0000 7f3644214640 10 monclient: init 2024-03-29T10:47:26.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.293+0000 7f3644214640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:26.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.293+0000 7f3644214640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:26.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.293+0000 7f3644214640 10 monclient(hunting): picked mon.noname-b con 0x7f363c0ad6c0 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:26.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.293+0000 7f3644214640 10 monclient(hunting): picked mon.noname-a con 0x7f363c0af6c0 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:26.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.294+0000 7f3644214640 10 monclient(hunting): picked mon.noname-c con 0x7f363c14fdc0 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:26.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.294+0000 7f3644214640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c0a94e0 con 0x7f363c0af6c0 2024-03-29T10:47:26.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.294+0000 7f3644214640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c0af3a0 con 0x7f363c14fdc0 2024-03-29T10:47:26.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.294+0000 7f3644214640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c0afb60 con 0x7f363c0ad6c0 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.294+0000 7f3644214640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.294+0000 7f3644214640 10 monclient(hunting): authenticate will time out at 1066.492029s 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.296+0000 7f36427b1640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f363c14fdc0 0x7f363c15a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:59801/0 (socket says 172.21.15.98:59801) 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.296+0000 7f36427b1640 1 -- 172.21.15.98:0/225927279 learned_addr learned my addr 172.21.15.98:0/225927279 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.297+0000 7f3640fae640 1 -- 172.21.15.98:0/225927279 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744669499 0 0) 0x7f363c0af3a0 con 0x7f363c14fdc0 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.297+0000 7f3640fae640 10 monclient(hunting): _negotiate 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.297+0000 7f3640fae640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.297+0000 7f3640fae640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.297+0000 7f3640fae640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.297+0000 7f3640fae640 1 -- 172.21.15.98:0/225927279 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3628003520 con 0x7f363c14fdc0 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.297+0000 7f3640fae640 1 -- 172.21.15.98:0/225927279 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4259612445 0 0) 0x7f3628003520 con 0x7f363c14fdc0 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.299+0000 7f3640fae640 1 -- 172.21.15.98:0/225927279 >> v1:172.21.15.98:6790/0 conn(0x7f363c0ad6c0 legacy=0x7f363c14f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.299+0000 7f3640fae640 1 -- 172.21.15.98:0/225927279 >> v1:172.21.15.98:6789/0 conn(0x7f363c0af6c0 legacy=0x7f363c152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:26.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.299+0000 7f3640fae640 10 monclient: _finish_hunting 0 2024-03-29T10:47:26.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.299+0000 7f3640fae640 1 monclient: found mon.noname-c 2024-03-29T10:47:26.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.299+0000 7f3640fae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:26.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.299+0000 7f3640fae640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:47:26.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.299+0000 7f3640fae640 1 -- 172.21.15.98:0/225927279 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c0aab80 con 0x7f363c14fdc0 2024-03-29T10:47:26.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.299+0000 7f3640fae640 10 monclient: _finish_auth 0 2024-03-29T10:47:26.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.299+0000 7f3640fae640 10 monclient: _check_auth_tickets 2024-03-29T10:47:26.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.299+0000 7f3640fae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:26.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.299+0000 7f3644214640 5 monclient: authenticate success, global_id 4324 2024-03-29T10:47:26.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.299+0000 7f3644214640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:26.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.299+0000 7f3640fae640 1 -- 172.21.15.98:0/225927279 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3634001a10 con 0x7f363c14fdc0 2024-03-29T10:47:26.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.300+0000 7f3640fae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:26.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.300+0000 7f3640fae640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:26.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.300+0000 7f3640fae640 10 monclient: dump: 2024-03-29T10:47:26.291 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:26.291 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:26.291 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:26.291 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:26.291 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:26.291 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:26.291 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:26.291 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:26.291 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:26.291 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:26.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.300+0000 7f3644214640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:26.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.301+0000 7f3640fae640 1 -- 172.21.15.98:0/225927279 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3634001c70 con 0x7f363c14fdc0 2024-03-29T10:47:26.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.301+0000 7f3640fae640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:26.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.301+0000 7f3640fae640 1 -- 172.21.15.98:0/225927279 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f36340020c0 con 0x7f363c14fdc0 2024-03-29T10:47:26.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.301+0000 7f3640fae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:26.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.301+0000 7f3640fae640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:26.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.301+0000 7f3640fae640 10 monclient: dump: 2024-03-29T10:47:26.293 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:26.294 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:26.294 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:26.294 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:26.295 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:26.295 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:26.295 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:26.296 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:26.296 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:26.296 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:26.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.301+0000 7f3644214640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:26.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.301+0000 7f3644214640 4 set_mon_vals no callback set 2024-03-29T10:47:26.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.302+0000 7f3644214640 10 monclient: shutdown 2024-03-29T10:47:26.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.302+0000 7f3644214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:26.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.302+0000 7f3644214640 1 -- 172.21.15.98:0/225927279 >> v1:172.21.15.189:6789/0 conn(0x7f363c14fdc0 legacy=0x7f363c15a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:26.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.304+0000 7f3644214640 1 -- 172.21.15.98:0/225927279 shutdown_connections 2024-03-29T10:47:26.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.304+0000 7f3644214640 1 -- 172.21.15.98:0/225927279 >> 172.21.15.98:0/225927279 conn(0x7f363c0b35e0 msgr2=0x7f363c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:26.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.305+0000 7f3644214640 1 -- 172.21.15.98:0/225927279 shutdown_connections 2024-03-29T10:47:26.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.305+0000 7f3644214640 1 -- 172.21.15.98:0/225927279 wait complete. 2024-03-29T10:47:26.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.306+0000 7f3644214640 10 monclient: build_initial_monmap 2024-03-29T10:47:26.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.306+0000 7f3644214640 10 monclient: monmap: 2024-03-29T10:47:26.306 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:26.306 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:26.306 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:47:26.306 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:47:26.306 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:26.306 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:26.307 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:26.307 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:47:26.307 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:47:26.307 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:26.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.306+0000 7f3644214640 1 librados: starting msgr at 2024-03-29T10:47:26.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.306+0000 7f3644214640 1 librados: starting objecter 2024-03-29T10:47:26.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.307+0000 7f3644214640 1 Processor -- start 2024-03-29T10:47:26.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.307+0000 7f3644214640 1 -- start start 2024-03-29T10:47:26.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.307+0000 7f3644214640 1 librados: setting wanted keys 2024-03-29T10:47:26.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.307+0000 7f3644214640 1 librados: calling monclient init 2024-03-29T10:47:26.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.307+0000 7f3644214640 10 monclient: init 2024-03-29T10:47:26.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.308+0000 7f3644214640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:26.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.308+0000 7f3644214640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:26.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.310+0000 7f3644214640 10 monclient(hunting): picked mon.noname-a con 0x7f363c0ad6c0 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:26.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.313+0000 7f3644214640 10 monclient(hunting): picked mon.noname-b con 0x7f363c0af6c0 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:26.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.313+0000 7f3641fb0640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f363c0ad6c0 0x7f363c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33023/0 (socket says 172.21.15.98:33023) 2024-03-29T10:47:26.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.313+0000 7f3644214640 10 monclient(hunting): picked mon.noname-c con 0x7f363c14fdc0 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:26.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.313+0000 7f3644214640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c13d5a0 con 0x7f363c0ad6c0 2024-03-29T10:47:26.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.313+0000 7f3644214640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c1e7460 con 0x7f363c0af6c0 2024-03-29T10:47:26.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.313+0000 7f3644214640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c0708a0 con 0x7f363c14fdc0 2024-03-29T10:47:26.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.313+0000 7f3644214640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:26.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.313+0000 7f3644214640 10 monclient(hunting): authenticate will time out at 1066.511954s 2024-03-29T10:47:26.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f3641fb0640 1 -- 172.21.15.98:0/1949507637 learned_addr learned my addr 172.21.15.98:0/1949507637 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:26.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 client.?.objecter ms_handle_connect 0x7f363c0ad6c0 2024-03-29T10:47:26.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:26.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550526177 0 0) 0x7f363c13d5a0 con 0x7f363c0ad6c0 2024-03-29T10:47:26.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:26.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:26.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:26.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:47:26.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3620004eb0 con 0x7f363c0ad6c0 2024-03-29T10:47:26.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 client.?.objecter ms_handle_connect 0x7f363c14fdc0 2024-03-29T10:47:26.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:26.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080065668 0 0) 0x7f363c0708a0 con 0x7f363c14fdc0 2024-03-29T10:47:26.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:26.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:26.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:26.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:47:26.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f363c13d5a0 con 0x7f363c14fdc0 2024-03-29T10:47:26.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2150756243 0 0) 0x7f3620004eb0 con 0x7f363c0ad6c0 2024-03-29T10:47:26.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f363c0708a0 con 0x7f363c0ad6c0 2024-03-29T10:47:26.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3638001960 con 0x7f363c0ad6c0 2024-03-29T10:47:26.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:26.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.314+0000 7f36327fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:26.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 10 monclient(hunting): dump: 2024-03-29T10:47:26.312 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:26.312 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:26.313 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:26.313 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:26.313 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:26.313 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:26.313 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:26.313 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:26.313 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:26.313 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:26.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:26.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2785897493 0 0) 0x7f363c13d5a0 con 0x7f363c14fdc0 2024-03-29T10:47:26.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3620004eb0 con 0x7f363c14fdc0 2024-03-29T10:47:26.313 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3634001830 con 0x7f363c14fdc0 2024-03-29T10:47:26.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:26.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:26.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 10 monclient(hunting): dump: 2024-03-29T10:47:26.314 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:26.314 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:26.314 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:26.314 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:26.314 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:26.315 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:26.315 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:26.315 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:26.315 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:26.315 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:26.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4291398636 0 0) 0x7f363c0708a0 con 0x7f363c0ad6c0 2024-03-29T10:47:26.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 >> v1:172.21.15.98:6790/0 conn(0x7f363c14fdc0 legacy=0x7f363c08a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:26.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 >> v1:172.21.15.189:6789/0 conn(0x7f363c0af6c0 legacy=0x7f363c1e7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:26.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:26.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 1 monclient: found mon.a 2024-03-29T10:47:26.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:26.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:26.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c06e7a0 con 0x7f363c0ad6c0 2024-03-29T10:47:26.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:26.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:26.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f36327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:26.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f3644214640 5 monclient: authenticate success, global_id 4311 2024-03-29T10:47:26.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f3644214640 10 monclient: _renew_subs 2024-03-29T10:47:26.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f3644214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:26.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f3644214640 1 -- 172.21.15.98:0/1949507637 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f363c1e8090 con 0x7f363c0ad6c0 2024-03-29T10:47:26.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f3644214640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:26.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f3644214640 10 monclient: _renew_subs 2024-03-29T10:47:26.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f3644214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:26.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f3644214640 1 -- 172.21.15.98:0/1949507637 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f363c070f80 con 0x7f363c0ad6c0 2024-03-29T10:47:26.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.315+0000 7f3644214640 1 librados: init done 2024-03-29T10:47:26.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.319+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36380035f0 con 0x7f363c0ad6c0 2024-03-29T10:47:26.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.319+0000 7f36327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:26.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.319+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3638003940 con 0x7f363c0ad6c0 2024-03-29T10:47:26.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.319+0000 7f36327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:26.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.319+0000 7f36327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:26.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.319+0000 7f3633fff640 4 set_mon_vals no callback set 2024-03-29T10:47:26.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.319+0000 7f36327fc640 10 monclient: dump: 2024-03-29T10:47:26.318 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:26.318 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:26.318 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.320+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f3638003b60 con 0x7f363c0ad6c0 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.320+0000 7f3644214640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.321+0000 7f3644214640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.321+0000 7f3644214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.321+0000 7f3644214640 1 -- 172.21.15.98:0/1949507637 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3624005080 con 0x7f363c0ad6c0 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.321+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f3638084a40 con 0x7f363c0ad6c0 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.321+0000 7f36327fc640 10 client.4311.objecter ms_dispatch 0x7f363c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:26.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.321+0000 7f36327fc640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:26.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.322+0000 7f36327fc640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:26.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.322+0000 7f36327fc640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-29T10:47:26.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.322+0000 7f36417af640 10 monclient: get_auth_request con 0x7f362006e030 auth_method 0 2024-03-29T10:47:26.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.322+0000 7f36327fc640 10 client.4311.objecter ms_handle_connect 0x7f362006e030 2024-03-29T10:47:26.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.332+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3638050320 con 0x7f363c0ad6c0 2024-03-29T10:47:26.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.332+0000 7f36327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:26.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.332+0000 7f36327fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:26.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.664+0000 7f3644214640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:26.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.664+0000 7f3644214640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:26.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.664+0000 7f3644214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:26.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.664+0000 7f3644214640 1 -- 172.21.15.98:0/1949507637 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3624005330 con 0x7f363c0ad6c0 2024-03-29T10:47:26.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.665+0000 7f36327fc640 1 -- 172.21.15.98:0/1949507637 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f3638052250 con 0x7f363c0ad6c0 2024-03-29T10:47:26.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.665+0000 7f36327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:26.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.665+0000 7f36327fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:26.650 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:47:26.651 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:26.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.670+0000 7f3644214640 10 librados: watch_flush enter 2024-03-29T10:47:26.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.670+0000 7f3644214640 10 librados: watch_flush exit 2024-03-29T10:47:26.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.670+0000 7f3644214640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-29T10:47:26.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.670+0000 7f3644214640 10 client.4311.objecter successfully canceled tick 2024-03-29T10:47:26.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.670+0000 7f3644214640 1 -- 172.21.15.98:0/1949507637 >> v1:172.21.15.98:6816/1481799426 conn(0x7f362006e030 legacy=0x7f36200704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:26.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.670+0000 7f3644214640 10 monclient: shutdown 2024-03-29T10:47:26.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.670+0000 7f3644214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:26.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.670+0000 7f3644214640 1 -- 172.21.15.98:0/1949507637 >> v1:172.21.15.98:6789/0 conn(0x7f363c0ad6c0 legacy=0x7f363c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:26.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.670+0000 7f3644214640 1 -- 172.21.15.98:0/1949507637 shutdown_connections 2024-03-29T10:47:26.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.670+0000 7f3644214640 1 -- 172.21.15.98:0/1949507637 >> 172.21.15.98:0/1949507637 conn(0x7f363c0b35e0 msgr2=0x7f363c0a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:26.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.671+0000 7f3644214640 1 -- 172.21.15.98:0/1949507637 shutdown_connections 2024-03-29T10:47:26.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.671+0000 7f3644214640 1 -- 172.21.15.98:0/1949507637 wait complete. 2024-03-29T10:47:26.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:26.672+0000 7f3644214640 1 librados: shutdown 2024-03-29T10:47:26.672 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:33.479 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:33.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.643+0000 7f2dc57d0640 10 monclient: get_monmap_and_config 2024-03-29T10:47:33.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.643+0000 7f2dc57d0640 10 monclient: build_initial_monmap 2024-03-29T10:47:33.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.643+0000 7f2dc57d0640 10 monclient: monmap: 2024-03-29T10:47:33.627 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:33.627 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:33.627 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:47:33.645164+0000 2024-03-29T10:47:33.627 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:47:33.645164+0000 2024-03-29T10:47:33.627 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:33.627 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:33.627 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:33.627 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:47:33.627 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:47:33.627 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:33.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.644+0000 7f2dc57d0640 1 Processor -- start 2024-03-29T10:47:33.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.644+0000 7f2dc57d0640 1 -- start start 2024-03-29T10:47:33.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.644+0000 7f2dc57d0640 10 monclient: init 2024-03-29T10:47:33.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.645+0000 7f2dc57d0640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:33.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.645+0000 7f2dc57d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:33.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.645+0000 7f2dc57d0640 10 monclient(hunting): picked mon.noname-c con 0x7f2dc00b1f60 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:33.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.645+0000 7f2dc57d0640 10 monclient(hunting): picked mon.noname-b con 0x7f2dc0071eb0 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:33.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.645+0000 7f2dc57d0640 10 monclient(hunting): picked mon.noname-a con 0x7f2dc0143d10 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:33.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.645+0000 7f2dc57d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc006ef80 con 0x7f2dc0143d10 2024-03-29T10:47:33.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.645+0000 7f2dc57d0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc006f260 con 0x7f2dc00b1f60 2024-03-29T10:47:33.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.645+0000 7f2dc57d0640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc00b40c0 con 0x7f2dc0071eb0 2024-03-29T10:47:33.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.645+0000 7f2dc57d0640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:33.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.645+0000 7f2dc57d0640 10 monclient(hunting): authenticate will time out at 1073.843259s 2024-03-29T10:47:33.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.648+0000 7f2dbf7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2dc0143d10 0x7f2dc0146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47495/0 (socket says 172.21.15.98:47495) 2024-03-29T10:47:33.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.648+0000 7f2dbf7fe640 1 -- 172.21.15.98:0/2896510288 learned_addr learned my addr 172.21.15.98:0/2896510288 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:33.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.649+0000 7f2dbdffb640 1 -- 172.21.15.98:0/2896510288 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566595983 0 0) 0x7f2dc006ef80 con 0x7f2dc0143d10 2024-03-29T10:47:33.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.649+0000 7f2dbdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:33.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.649+0000 7f2dbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:33.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.649+0000 7f2dbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:33.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.649+0000 7f2dbdffb640 10 monclient(hunting): my global_id is 4314 2024-03-29T10:47:33.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.649+0000 7f2dbdffb640 1 -- 172.21.15.98:0/2896510288 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da8003520 con 0x7f2dc0143d10 2024-03-29T10:47:33.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.649+0000 7f2dbdffb640 1 -- 172.21.15.98:0/2896510288 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2960948860 0 0) 0x7f2da8003520 con 0x7f2dc0143d10 2024-03-29T10:47:33.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.649+0000 7f2dbdffb640 1 -- 172.21.15.98:0/2896510288 >> v1:172.21.15.98:6790/0 conn(0x7f2dc0071eb0 legacy=0x7f2dc0146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:33.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 1 -- 172.21.15.98:0/2896510288 >> v1:172.21.15.189:6789/0 conn(0x7f2dc00b1f60 legacy=0x7f2dc0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:33.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:33.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 1 monclient: found mon.noname-a 2024-03-29T10:47:33.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:33.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:47:33.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 1 -- 172.21.15.98:0/2896510288 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc0155d70 con 0x7f2dc0143d10 2024-03-29T10:47:33.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:33.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:33.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:33.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 1 -- 172.21.15.98:0/2896510288 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2db8001a10 con 0x7f2dc0143d10 2024-03-29T10:47:33.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:33.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:33.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 10 monclient: dump: 2024-03-29T10:47:33.634 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:33.634 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:33.634 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:33.634 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:33.634 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:33.634 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 1 -- 172.21.15.98:0/2896510288 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2db8001ca0 con 0x7f2dc0143d10 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 1 -- 172.21.15.98:0/2896510288 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2db8002030 con 0x7f2dc0143d10 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.650+0000 7f2dbdffb640 10 monclient: dump: 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:33.635 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:33.636 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:33.636 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:33.636 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:33.636 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:33.636 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:33.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.651+0000 7f2dc57d0640 5 monclient: authenticate success, global_id 4314 2024-03-29T10:47:33.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.651+0000 7f2dc57d0640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:33.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.651+0000 7f2dc57d0640 4 set_mon_vals no callback set 2024-03-29T10:47:33.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.652+0000 7f2dc57d0640 10 monclient: shutdown 2024-03-29T10:47:33.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.652+0000 7f2dc57d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:33.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.652+0000 7f2dc57d0640 1 -- 172.21.15.98:0/2896510288 >> v1:172.21.15.98:6789/0 conn(0x7f2dc0143d10 legacy=0x7f2dc0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:33.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.652+0000 7f2dc57d0640 1 -- 172.21.15.98:0/2896510288 shutdown_connections 2024-03-29T10:47:33.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.652+0000 7f2dc57d0640 1 -- 172.21.15.98:0/2896510288 >> 172.21.15.98:0/2896510288 conn(0x7f2dc0066a00 msgr2=0x7f2dc0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:33.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.654+0000 7f2dc57d0640 1 -- 172.21.15.98:0/2896510288 shutdown_connections 2024-03-29T10:47:33.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.654+0000 7f2dc57d0640 1 -- 172.21.15.98:0/2896510288 wait complete. 2024-03-29T10:47:33.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.655+0000 7f2dc57d0640 10 monclient: build_initial_monmap 2024-03-29T10:47:33.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.655+0000 7f2dc57d0640 10 monclient: monmap: 2024-03-29T10:47:33.649 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:33.649 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:33.649 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:47:33.649 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:47:33.649 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:33.649 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:33.649 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:33.649 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:47:33.650 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:47:33.650 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:33.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.656+0000 7f2dc57d0640 1 librados: starting msgr at 2024-03-29T10:47:33.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.656+0000 7f2dc57d0640 1 librados: starting objecter 2024-03-29T10:47:33.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.656+0000 7f2dc57d0640 1 Processor -- start 2024-03-29T10:47:33.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.656+0000 7f2dc57d0640 1 -- start start 2024-03-29T10:47:33.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.656+0000 7f2dc57d0640 1 librados: setting wanted keys 2024-03-29T10:47:33.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.656+0000 7f2dc57d0640 1 librados: calling monclient init 2024-03-29T10:47:33.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.656+0000 7f2dc57d0640 10 monclient: init 2024-03-29T10:47:33.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.657+0000 7f2dc57d0640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:33.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.657+0000 7f2dc57d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:33.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.657+0000 7f2dc57d0640 10 monclient(hunting): picked mon.noname-c con 0x7f2dc00b1f60 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:33.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.657+0000 7f2dc57d0640 10 monclient(hunting): picked mon.noname-a con 0x7f2dc0143d10 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:33.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.657+0000 7f2dc57d0640 10 monclient(hunting): picked mon.noname-b con 0x7f2dc01378e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:33.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.657+0000 7f2dc57d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc0139560 con 0x7f2dc0143d10 2024-03-29T10:47:33.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.657+0000 7f2dc57d0640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc01397d0 con 0x7f2dc01378e0 2024-03-29T10:47:33.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.657+0000 7f2dc57d0640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc0141090 con 0x7f2dc00b1f60 2024-03-29T10:47:33.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.657+0000 7f2dc57d0640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:33.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.657+0000 7f2dc57d0640 10 monclient(hunting): authenticate will time out at 1073.855251s 2024-03-29T10:47:33.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.663+0000 7f2dbf7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2dc01378e0 0x7f2dc013fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:42543/0 (socket says 172.21.15.98:42543) 2024-03-29T10:47:33.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.663+0000 7f2dbf7fe640 1 -- 172.21.15.98:0/249604666 learned_addr learned my addr 172.21.15.98:0/249604666 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:33.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.663+0000 7f2da77fe640 10 client.?.objecter ms_handle_connect 0x7f2dc01378e0 2024-03-29T10:47:33.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.663+0000 7f2da77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:33.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.663+0000 7f2da77fe640 10 client.?.objecter ms_handle_connect 0x7f2dc0143d10 2024-03-29T10:47:33.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.664+0000 7f2da77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:33.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.666+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649212118 0 0) 0x7f2dc0139560 con 0x7f2dc0143d10 2024-03-29T10:47:33.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.666+0000 7f2da77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:33.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.666+0000 7f2da77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:33.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.666+0000 7f2da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:33.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.666+0000 7f2da77fe640 10 monclient(hunting): my global_id is 4317 2024-03-29T10:47:33.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.666+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dac004eb0 con 0x7f2dc0143d10 2024-03-29T10:47:33.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.666+0000 7f2da77fe640 10 client.?.objecter ms_handle_connect 0x7f2dc00b1f60 2024-03-29T10:47:33.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.666+0000 7f2da77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:33.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.666+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2295123411 0 0) 0x7f2dac004eb0 con 0x7f2dc0143d10 2024-03-29T10:47:33.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dc0139560 con 0x7f2dc0143d10 2024-03-29T10:47:33.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2db4001870 con 0x7f2dc0143d10 2024-03-29T10:47:33.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:33.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:33.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 10 monclient(hunting): dump: 2024-03-29T10:47:33.654 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:33.654 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:33.654 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:33.654 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:33.654 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:33.654 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:33.654 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:33.654 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:33.654 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:33.655 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:33.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:33.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690473913 0 0) 0x7f2dc0141090 con 0x7f2dc00b1f60 2024-03-29T10:47:33.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:33.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:33.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:33.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:47:33.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dac004eb0 con 0x7f2dc00b1f60 2024-03-29T10:47:33.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598587910 0 0) 0x7f2dc01397d0 con 0x7f2dc01378e0 2024-03-29T10:47:33.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:33.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:33.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:33.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 10 monclient(hunting): my global_id is 4330 2024-03-29T10:47:33.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.669+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc0141090 con 0x7f2dc01378e0 2024-03-29T10:47:33.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1788290543 0 0) 0x7f2dc0139560 con 0x7f2dc0143d10 2024-03-29T10:47:33.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 >> v1:172.21.15.98:6790/0 conn(0x7f2dc00b1f60 legacy=0x7f2dc0141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:33.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 >> v1:172.21.15.189:6789/0 conn(0x7f2dc01378e0 legacy=0x7f2dc013fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:33.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:33.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 1 monclient: found mon.a 2024-03-29T10:47:33.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:33.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:33.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc0141370 con 0x7f2dc0143d10 2024-03-29T10:47:33.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:33.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:33.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:33.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2dc57d0640 5 monclient: authenticate success, global_id 4317 2024-03-29T10:47:33.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2dc57d0640 10 monclient: _renew_subs 2024-03-29T10:47:33.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2dc57d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:33.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2dc57d0640 1 -- 172.21.15.98:0/249604666 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc0141fa0 con 0x7f2dc0143d10 2024-03-29T10:47:33.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2dc57d0640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:33.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2dc57d0640 10 monclient: _renew_subs 2024-03-29T10:47:33.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2dc57d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:33.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2dc57d0640 1 -- 172.21.15.98:0/249604666 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc0070920 con 0x7f2dc0143d10 2024-03-29T10:47:33.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2dc57d0640 1 librados: init done 2024-03-29T10:47:33.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2db40035f0 con 0x7f2dc0143d10 2024-03-29T10:47:33.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:33.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2db4003900 con 0x7f2dc0143d10 2024-03-29T10:47:33.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:33.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:33.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2da77fe640 10 monclient: dump: 2024-03-29T10:47:33.658 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:33.658 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:33.658 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:33.658 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:33.658 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:33.658 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:33.658 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:33.659 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:33.659 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:33.659 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:33.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.670+0000 7f2dbd7fa640 4 set_mon_vals no callback set 2024-03-29T10:47:33.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.672+0000 7f2dc57d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:33.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.672+0000 7f2dc57d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:33.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.672+0000 7f2dc57d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:33.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.672+0000 7f2dc57d0640 1 -- 172.21.15.98:0/249604666 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da0005080 con 0x7f2dc0143d10 2024-03-29T10:47:33.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.675+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f2db40198d0 con 0x7f2dc0143d10 2024-03-29T10:47:33.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.676+0000 7f2dbeffd640 10 monclient: get_auth_request con 0x7f2dac06e040 auth_method 0 2024-03-29T10:47:33.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.676+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f2db40534f0 con 0x7f2dc0143d10 2024-03-29T10:47:33.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.676+0000 7f2da77fe640 10 client.4317.objecter ms_dispatch 0x7f2dc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:33.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.676+0000 7f2da77fe640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:33.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.676+0000 7f2da77fe640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:33.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.676+0000 7f2da77fe640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-29T10:47:33.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.676+0000 7f2da77fe640 10 client.4317.objecter ms_handle_connect 0x7f2dac06e040 2024-03-29T10:47:33.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.686+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2db404f790 con 0x7f2dc0143d10 2024-03-29T10:47:33.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.686+0000 7f2da77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:33.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:33.686+0000 7f2da77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:33.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.015+0000 7f2dc57d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:33.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.015+0000 7f2dc57d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:33.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.015+0000 7f2dc57d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:33.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.015+0000 7f2dc57d0640 1 -- 172.21.15.98:0/249604666 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2da0005330 con 0x7f2dc0143d10 2024-03-29T10:47:33.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.016+0000 7f2da77fe640 1 -- 172.21.15.98:0/249604666 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f2db404f790 con 0x7f2dc0143d10 2024-03-29T10:47:33.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.016+0000 7f2da77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:33.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.016+0000 7f2da77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:34.000 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:47:34.000 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:34.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.019+0000 7f2dc57d0640 10 librados: watch_flush enter 2024-03-29T10:47:34.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.020+0000 7f2dc57d0640 10 librados: watch_flush exit 2024-03-29T10:47:34.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.020+0000 7f2dc57d0640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-29T10:47:34.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.020+0000 7f2dc57d0640 10 client.4317.objecter successfully canceled tick 2024-03-29T10:47:34.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.023+0000 7f2dc57d0640 1 -- 172.21.15.98:0/249604666 >> v1:172.21.15.98:6816/1481799426 conn(0x7f2dac06e040 legacy=0x7f2dac070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:34.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.023+0000 7f2dc57d0640 10 monclient: shutdown 2024-03-29T10:47:34.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.023+0000 7f2dc57d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:34.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.023+0000 7f2dc57d0640 1 -- 172.21.15.98:0/249604666 >> v1:172.21.15.98:6789/0 conn(0x7f2dc0143d10 legacy=0x7f2dc0134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:34.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.023+0000 7f2dc57d0640 1 -- 172.21.15.98:0/249604666 shutdown_connections 2024-03-29T10:47:34.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.023+0000 7f2dc57d0640 1 -- 172.21.15.98:0/249604666 >> 172.21.15.98:0/249604666 conn(0x7f2dc0066a00 msgr2=0x7f2dc00b7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:34.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.023+0000 7f2dc57d0640 1 -- 172.21.15.98:0/249604666 shutdown_connections 2024-03-29T10:47:34.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.023+0000 7f2dc57d0640 1 -- 172.21.15.98:0/249604666 wait complete. 2024-03-29T10:47:34.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:34.023+0000 7f2dc57d0640 1 librados: shutdown 2024-03-29T10:47:34.022 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:40.830 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:40.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.001+0000 7f7963dcb640 10 monclient: get_monmap_and_config 2024-03-29T10:47:40.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.001+0000 7f7963dcb640 10 monclient: build_initial_monmap 2024-03-29T10:47:40.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.001+0000 7f7963dcb640 10 monclient: monmap: 2024-03-29T10:47:40.984 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:40.984 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:40.984 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:47:41.002607+0000 2024-03-29T10:47:40.985 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:47:41.002607+0000 2024-03-29T10:47:40.985 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:40.985 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:40.985 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:40.985 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:47:40.985 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:47:40.985 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:40.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.003+0000 7f7963dcb640 1 Processor -- start 2024-03-29T10:47:40.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.003+0000 7f7963dcb640 1 -- start start 2024-03-29T10:47:40.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.003+0000 7f7963dcb640 10 monclient: init 2024-03-29T10:47:40.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.004+0000 7f7963dcb640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:40.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.004+0000 7f7963dcb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:40.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.004+0000 7f7963dcb640 10 monclient(hunting): picked mon.noname-a con 0x7f795c0a9750 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:40.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.004+0000 7f7963dcb640 10 monclient(hunting): picked mon.noname-c con 0x7f795c13e1e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:40.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.004+0000 7f7963dcb640 10 monclient(hunting): picked mon.noname-b con 0x7f795c071c30 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:40.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.004+0000 7f7963dcb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c0ab300 con 0x7f795c0a9750 2024-03-29T10:47:40.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.004+0000 7f7963dcb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c0ac2f0 con 0x7f795c13e1e0 2024-03-29T10:47:40.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.004+0000 7f7963dcb640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c0ad8d0 con 0x7f795c071c30 2024-03-29T10:47:40.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.004+0000 7f7963dcb640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:40.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.004+0000 7f7963dcb640 10 monclient(hunting): authenticate will time out at 1081.201360s 2024-03-29T10:47:40.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.010+0000 7f7962368640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f795c071c30 0x7f795c0ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:50495/0 (socket says 172.21.15.98:50495) 2024-03-29T10:47:40.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.010+0000 7f7962368640 1 -- 172.21.15.98:0/484865084 learned_addr learned my addr 172.21.15.98:0/484865084 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:40.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.011+0000 7f7960b65640 1 -- 172.21.15.98:0/484865084 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956597238 0 0) 0x7f795c0ad8d0 con 0x7f795c071c30 2024-03-29T10:47:40.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.011+0000 7f7960b65640 10 monclient(hunting): _negotiate 2024-03-29T10:47:40.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.011+0000 7f7960b65640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:40.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.011+0000 7f7960b65640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:40.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.011+0000 7f7960b65640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:47:40.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.011+0000 7f7960b65640 1 -- 172.21.15.98:0/484865084 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7948003520 con 0x7f795c071c30 2024-03-29T10:47:40.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 1 -- 172.21.15.98:0/484865084 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734303609 0 0) 0x7f795c0ac2f0 con 0x7f795c13e1e0 2024-03-29T10:47:40.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 10 monclient(hunting): _negotiate 2024-03-29T10:47:40.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:40.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:40.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 10 monclient(hunting): my global_id is 4333 2024-03-29T10:47:40.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 1 -- 172.21.15.98:0/484865084 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f795c0ad8d0 con 0x7f795c13e1e0 2024-03-29T10:47:40.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 1 -- 172.21.15.98:0/484865084 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 486995713 0 0) 0x7f7948003520 con 0x7f795c071c30 2024-03-29T10:47:40.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 1 -- 172.21.15.98:0/484865084 >> v1:172.21.15.189:6789/0 conn(0x7f795c13e1e0 legacy=0x7f795c0ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:40.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 1 -- 172.21.15.98:0/484865084 >> v1:172.21.15.98:6789/0 conn(0x7f795c0a9750 legacy=0x7f795c13dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:40.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 10 monclient: _finish_hunting 0 2024-03-29T10:47:40.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 1 monclient: found mon.noname-b 2024-03-29T10:47:40.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:40.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:47:40.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 1 -- 172.21.15.98:0/484865084 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c0ada60 con 0x7f795c071c30 2024-03-29T10:47:40.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 10 monclient: _finish_auth 0 2024-03-29T10:47:40.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 10 monclient: _check_auth_tickets 2024-03-29T10:47:40.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:40.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7963dcb640 5 monclient: authenticate success, global_id 4328 2024-03-29T10:47:40.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7963dcb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:40.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 1 -- 172.21.15.98:0/484865084 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f794c001690 con 0x7f795c071c30 2024-03-29T10:47:40.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:40.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:40.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7960b65640 10 monclient: dump: 2024-03-29T10:47:40.999 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:40.999 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:40.999 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:40.999 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:40.999 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:40.999 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:40.999 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:40.999 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:41.000 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:41.000 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:41.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.014+0000 7f7963dcb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:41.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.017+0000 7f7960b65640 1 -- 172.21.15.98:0/484865084 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f794c0018b0 con 0x7f795c071c30 2024-03-29T10:47:41.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.018+0000 7f7960b65640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:41.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.018+0000 7f7963dcb640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:41.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.018+0000 7f7963dcb640 4 set_mon_vals no callback set 2024-03-29T10:47:41.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.018+0000 7f7960b65640 1 -- 172.21.15.98:0/484865084 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f794c001c40 con 0x7f795c071c30 2024-03-29T10:47:41.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.018+0000 7f7960b65640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:41.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.018+0000 7f7960b65640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:41.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.018+0000 7f7960b65640 10 monclient: dump: 2024-03-29T10:47:41.002 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:41.002 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:41.002 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:41.002 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:41.002 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:41.002 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:41.002 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:41.002 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:41.002 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:41.002 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:41.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.018+0000 7f7963dcb640 10 monclient: shutdown 2024-03-29T10:47:41.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.018+0000 7f7963dcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:41.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.018+0000 7f7963dcb640 1 -- 172.21.15.98:0/484865084 >> v1:172.21.15.98:6790/0 conn(0x7f795c071c30 legacy=0x7f795c0ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:41.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.019+0000 7f7963dcb640 1 -- 172.21.15.98:0/484865084 shutdown_connections 2024-03-29T10:47:41.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.019+0000 7f7963dcb640 1 -- 172.21.15.98:0/484865084 >> 172.21.15.98:0/484865084 conn(0x7f795c06b660 msgr2=0x7f795c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:41.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.023+0000 7f7963dcb640 1 -- 172.21.15.98:0/484865084 shutdown_connections 2024-03-29T10:47:41.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.023+0000 7f7963dcb640 1 -- 172.21.15.98:0/484865084 wait complete. 2024-03-29T10:47:41.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.024+0000 7f7963dcb640 10 monclient: build_initial_monmap 2024-03-29T10:47:41.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.024+0000 7f7963dcb640 10 monclient: monmap: 2024-03-29T10:47:41.007 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:41.007 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:41.007 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:47:41.008 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:47:41.008 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:41.008 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:41.008 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:41.008 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:47:41.008 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:47:41.008 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:41.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.025+0000 7f7963dcb640 1 librados: starting msgr at 2024-03-29T10:47:41.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.025+0000 7f7963dcb640 1 librados: starting objecter 2024-03-29T10:47:41.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.025+0000 7f7963dcb640 1 Processor -- start 2024-03-29T10:47:41.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.025+0000 7f7963dcb640 1 -- start start 2024-03-29T10:47:41.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.025+0000 7f7963dcb640 1 librados: setting wanted keys 2024-03-29T10:47:41.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.025+0000 7f7963dcb640 1 librados: calling monclient init 2024-03-29T10:47:41.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.025+0000 7f7963dcb640 10 monclient: init 2024-03-29T10:47:41.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.026+0000 7f7963dcb640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:41.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.026+0000 7f7963dcb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:41.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.026+0000 7f7963dcb640 10 monclient(hunting): picked mon.noname-a con 0x7f795c071c30 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:41.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.026+0000 7f7963dcb640 10 monclient(hunting): picked mon.noname-b con 0x7f795c0a9750 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:41.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.026+0000 7f7963dcb640 10 monclient(hunting): picked mon.noname-c con 0x7f795c13e1e0 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:41.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.026+0000 7f7963dcb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c141270 con 0x7f795c071c30 2024-03-29T10:47:41.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.026+0000 7f7963dcb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c1414e0 con 0x7f795c0a9750 2024-03-29T10:47:41.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.026+0000 7f7963dcb640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c1417c0 con 0x7f795c13e1e0 2024-03-29T10:47:41.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.026+0000 7f7963dcb640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:41.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.026+0000 7f7963dcb640 10 monclient(hunting): authenticate will time out at 1081.223470s 2024-03-29T10:47:41.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.031+0000 7f7962368640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f795c13e1e0 0x7f795c1efb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:50629/0 (socket says 172.21.15.98:50629) 2024-03-29T10:47:41.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.031+0000 7f7962368640 1 -- 172.21.15.98:0/3613918664 learned_addr learned my addr 172.21.15.98:0/3613918664 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:41.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.032+0000 7f79527fc640 10 client.?.objecter ms_handle_connect 0x7f795c13e1e0 2024-03-29T10:47:41.014 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.032+0000 7f79527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:41.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.032+0000 7f79527fc640 10 client.?.objecter ms_handle_connect 0x7f795c0a9750 2024-03-29T10:47:41.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:41.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 client.?.objecter ms_handle_connect 0x7f795c071c30 2024-03-29T10:47:41.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:41.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024107984 0 0) 0x7f795c1414e0 con 0x7f795c0a9750 2024-03-29T10:47:41.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:41.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:41.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:41.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:47:41.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7940004eb0 con 0x7f795c0a9750 2024-03-29T10:47:41.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471303191 0 0) 0x7f795c141270 con 0x7f795c071c30 2024-03-29T10:47:41.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:41.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:41.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:41.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 monclient(hunting): my global_id is 4323 2024-03-29T10:47:41.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f795c1414e0 con 0x7f795c071c30 2024-03-29T10:47:41.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 153746129 0 0) 0x7f795c1414e0 con 0x7f795c071c30 2024-03-29T10:47:41.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f795c141270 con 0x7f795c071c30 2024-03-29T10:47:41.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f79580018b0 con 0x7f795c071c30 2024-03-29T10:47:41.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:41.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:41.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 monclient(hunting): dump: 2024-03-29T10:47:41.024 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:41.024 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:41.024 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:41.024 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:41.024 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:41.024 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:41.024 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:41.024 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:41.024 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:41.024 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:41.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.033+0000 7f79527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:41.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1205997597 0 0) 0x7f795c141270 con 0x7f795c071c30 2024-03-29T10:47:41.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 >> v1:172.21.15.98:6790/0 conn(0x7f795c13e1e0 legacy=0x7f795c1efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:41.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 >> v1:172.21.15.189:6789/0 conn(0x7f795c0a9750 legacy=0x7f795c148370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:41.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:41.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 1 monclient: found mon.a 2024-03-29T10:47:41.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:41.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:41.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c141aa0 con 0x7f795c071c30 2024-03-29T10:47:41.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:41.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:41.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:41.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f7963dcb640 5 monclient: authenticate success, global_id 4323 2024-03-29T10:47:41.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7958003610 con 0x7f795c071c30 2024-03-29T10:47:41.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:41.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f7963dcb640 10 monclient: _renew_subs 2024-03-29T10:47:41.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f7958003a60 con 0x7f795c071c30 2024-03-29T10:47:41.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f7963dcb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:41.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f7963dcb640 1 -- 172.21.15.98:0/3613918664 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795c140080 con 0x7f795c071c30 2024-03-29T10:47:41.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f7963dcb640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:41.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f7963dcb640 10 monclient: _renew_subs 2024-03-29T10:47:41.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f7963dcb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:41.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f7963dcb640 1 -- 172.21.15.98:0/3613918664 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f795c071670 con 0x7f795c071c30 2024-03-29T10:47:41.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f7963dcb640 1 librados: init done 2024-03-29T10:47:41.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79537fe640 4 set_mon_vals no callback set 2024-03-29T10:47:41.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:41.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:41.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.034+0000 7f79527fc640 10 monclient: dump: 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.037+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f7958003bc0 con 0x7f795c071c30 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.037+0000 7f7961366640 10 monclient: get_auth_request con 0x7f794006ddb0 auth_method 0 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.038+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f79580836c0 con 0x7f795c071c30 2024-03-29T10:47:41.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.038+0000 7f79527fc640 10 client.4323.objecter ms_dispatch 0x7f795c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:41.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.038+0000 7f79527fc640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:41.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.038+0000 7f79527fc640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:41.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.038+0000 7f79527fc640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-29T10:47:41.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.039+0000 7f79527fc640 10 client.4323.objecter ms_handle_connect 0x7f794006ddb0 2024-03-29T10:47:41.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.039+0000 7f7963dcb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:41.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.039+0000 7f7963dcb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:41.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.039+0000 7f7963dcb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:41.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.039+0000 7f7963dcb640 1 -- 172.21.15.98:0/3613918664 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f795c0abb00 con 0x7f795c071c30 2024-03-29T10:47:41.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.047+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f795804efa0 con 0x7f795c071c30 2024-03-29T10:47:41.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.048+0000 7f79527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:41.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.048+0000 7f79527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:41.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.379+0000 7f7963dcb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:41.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.379+0000 7f7963dcb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:41.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.379+0000 7f7963dcb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:47:41.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.379+0000 7f7963dcb640 1 -- 172.21.15.98:0/3613918664 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f795c075f10 con 0x7f795c071c30 2024-03-29T10:47:41.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.381+0000 7f79527fc640 1 -- 172.21.15.98:0/3613918664 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f7958050ed0 con 0x7f795c071c30 2024-03-29T10:47:41.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.381+0000 7f79527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:41.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.381+0000 7f79527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:41.366 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:47:41.366 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:41.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.386+0000 7f7963dcb640 10 librados: watch_flush enter 2024-03-29T10:47:41.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.386+0000 7f7963dcb640 10 librados: watch_flush exit 2024-03-29T10:47:41.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.386+0000 7f7963dcb640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-29T10:47:41.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.386+0000 7f7963dcb640 10 client.4323.objecter successfully canceled tick 2024-03-29T10:47:41.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.386+0000 7f7963dcb640 1 -- 172.21.15.98:0/3613918664 >> v1:172.21.15.98:6816/1481799426 conn(0x7f794006ddb0 legacy=0x7f7940070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:41.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.386+0000 7f7963dcb640 10 monclient: shutdown 2024-03-29T10:47:41.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.386+0000 7f7963dcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:41.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.386+0000 7f7963dcb640 1 -- 172.21.15.98:0/3613918664 >> v1:172.21.15.98:6789/0 conn(0x7f795c071c30 legacy=0x7f795c13f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:41.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.386+0000 7f7963dcb640 1 -- 172.21.15.98:0/3613918664 shutdown_connections 2024-03-29T10:47:41.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.386+0000 7f7963dcb640 1 -- 172.21.15.98:0/3613918664 >> 172.21.15.98:0/3613918664 conn(0x7f795c06b660 msgr2=0x7f795c0b3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:41.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.388+0000 7f7963dcb640 1 -- 172.21.15.98:0/3613918664 shutdown_connections 2024-03-29T10:47:41.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.389+0000 7f7963dcb640 1 -- 172.21.15.98:0/3613918664 wait complete. 2024-03-29T10:47:41.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:41.389+0000 7f7963dcb640 1 librados: shutdown 2024-03-29T10:47:41.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:48.200 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:48.349 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.367+0000 7f438fd32640 10 monclient: get_monmap_and_config 2024-03-29T10:47:48.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.367+0000 7f438fd32640 10 monclient: build_initial_monmap 2024-03-29T10:47:48.350 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.367+0000 7f438fd32640 10 monclient: monmap: 2024-03-29T10:47:48.350 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:48.350 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:48.350 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:47:48.368493+0000 2024-03-29T10:47:48.350 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:47:48.368493+0000 2024-03-29T10:47:48.350 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:48.350 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:48.350 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:48.350 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:47:48.350 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:47:48.350 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:48.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.368+0000 7f438fd32640 1 Processor -- start 2024-03-29T10:47:48.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.368+0000 7f438fd32640 1 -- start start 2024-03-29T10:47:48.351 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.368+0000 7f438fd32640 10 monclient: init 2024-03-29T10:47:48.353 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.369+0000 7f438fd32640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:48.353 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.369+0000 7f438fd32640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:48.353 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.369+0000 7f438fd32640 10 monclient(hunting): picked mon.noname-c con 0x7f43880b0860 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:48.353 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.370+0000 7f438fd32640 10 monclient(hunting): picked mon.noname-a con 0x7f4388072450 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:48.353 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.370+0000 7f438fd32640 10 monclient(hunting): picked mon.noname-b con 0x7f438813ee40 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:48.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.370+0000 7f438fd32640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43880ad840 con 0x7f4388072450 2024-03-29T10:47:48.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.370+0000 7f438fd32640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438813c830 con 0x7f43880b0860 2024-03-29T10:47:48.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.370+0000 7f438fd32640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438813c9c0 con 0x7f438813ee40 2024-03-29T10:47:48.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.370+0000 7f438fd32640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:48.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.370+0000 7f438fd32640 10 monclient(hunting): authenticate will time out at 1088.567115s 2024-03-29T10:47:48.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.370+0000 7f438dace640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f43880b0860 0x7f43880acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:48672/0 (socket says 172.21.15.98:48672) 2024-03-29T10:47:48.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.370+0000 7f438e2cf640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f438813ee40 0x7f438813f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:44902/0 (socket says 172.21.15.98:44902) 2024-03-29T10:47:48.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.370+0000 7f438dace640 1 -- 172.21.15.98:0/3488162766 learned_addr learned my addr 172.21.15.98:0/3488162766 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:48.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 1 -- 172.21.15.98:0/3488162766 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567021823 0 0) 0x7f438813c830 con 0x7f43880b0860 2024-03-29T10:47:48.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:48.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:48.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:48.354 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 10 monclient(hunting): my global_id is 4339 2024-03-29T10:47:48.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 1 -- 172.21.15.98:0/3488162766 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4374003520 con 0x7f43880b0860 2024-03-29T10:47:48.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 1 -- 172.21.15.98:0/3488162766 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727384328 0 0) 0x7f43880ad840 con 0x7f4388072450 2024-03-29T10:47:48.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:48.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:48.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:48.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:47:48.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 1 -- 172.21.15.98:0/3488162766 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f438813c830 con 0x7f4388072450 2024-03-29T10:47:48.355 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 1 -- 172.21.15.98:0/3488162766 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861957150 0 0) 0x7f438813c9c0 con 0x7f438813ee40 2024-03-29T10:47:48.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:48.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:48.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:48.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:47:48.356 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 1 -- 172.21.15.98:0/3488162766 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43880ad840 con 0x7f438813ee40 2024-03-29T10:47:48.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.371+0000 7f438cacc640 1 -- 172.21.15.98:0/3488162766 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 280329934 0 0) 0x7f438813c830 con 0x7f4388072450 2024-03-29T10:47:48.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 1 -- 172.21.15.98:0/3488162766 >> v1:172.21.15.98:6790/0 conn(0x7f438813ee40 legacy=0x7f438813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 1 -- 172.21.15.98:0/3488162766 >> v1:172.21.15.189:6789/0 conn(0x7f43880b0860 legacy=0x7f43880acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:48.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 1 monclient: found mon.noname-a 2024-03-29T10:47:48.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:48.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:47:48.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 1 -- 172.21.15.98:0/3488162766 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438813f970 con 0x7f4388072450 2024-03-29T10:47:48.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 10 monclient: _finish_auth 0 2024-03-29T10:47:48.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:48.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:48.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438fd32640 5 monclient: authenticate success, global_id 4326 2024-03-29T10:47:48.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438fd32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:48.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 1 -- 172.21.15.98:0/3488162766 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f43780015f0 con 0x7f4388072450 2024-03-29T10:47:48.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:48.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:48.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 10 monclient: dump: 2024-03-29T10:47:48.365 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:48.365 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:48.365 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:48.365 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:48.365 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:48.365 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:48.365 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:48.366 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:48.366 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:48.366 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:48.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438fd32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:48.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 1 -- 172.21.15.98:0/3488162766 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4378001750 con 0x7f4388072450 2024-03-29T10:47:48.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:48.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438fd32640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:48.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438fd32640 4 set_mon_vals no callback set 2024-03-29T10:47:48.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 1 -- 172.21.15.98:0/3488162766 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4378001b60 con 0x7f4388072450 2024-03-29T10:47:48.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:48.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:48.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.372+0000 7f438cacc640 10 monclient: dump: 2024-03-29T10:47:48.366 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:48.367 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:48.367 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:48.367 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:48.367 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:48.367 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:48.367 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:48.367 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:48.367 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:48.367 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:48.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.373+0000 7f438fd32640 10 monclient: shutdown 2024-03-29T10:47:48.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.373+0000 7f438fd32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:48.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.373+0000 7f438fd32640 1 -- 172.21.15.98:0/3488162766 >> v1:172.21.15.98:6789/0 conn(0x7f4388072450 legacy=0x7f438813c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.373+0000 7f438fd32640 1 -- 172.21.15.98:0/3488162766 shutdown_connections 2024-03-29T10:47:48.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.373+0000 7f438fd32640 1 -- 172.21.15.98:0/3488162766 >> 172.21.15.98:0/3488162766 conn(0x7f438806b660 msgr2=0x7f438806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:48.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.373+0000 7f438fd32640 1 -- 172.21.15.98:0/3488162766 shutdown_connections 2024-03-29T10:47:48.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.373+0000 7f438fd32640 1 -- 172.21.15.98:0/3488162766 wait complete. 2024-03-29T10:47:48.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.374+0000 7f438fd32640 10 monclient: build_initial_monmap 2024-03-29T10:47:48.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.374+0000 7f438fd32640 10 monclient: monmap: 2024-03-29T10:47:48.368 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:48.368 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:48.369 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:47:48.369 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:47:48.369 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:48.369 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:48.369 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:48.369 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:47:48.369 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:47:48.369 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:48.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.374+0000 7f438fd32640 1 librados: starting msgr at 2024-03-29T10:47:48.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.374+0000 7f438fd32640 1 librados: starting objecter 2024-03-29T10:47:48.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.375+0000 7f438fd32640 1 Processor -- start 2024-03-29T10:47:48.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.375+0000 7f438fd32640 1 -- start start 2024-03-29T10:47:48.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.375+0000 7f438fd32640 1 librados: setting wanted keys 2024-03-29T10:47:48.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.375+0000 7f438fd32640 1 librados: calling monclient init 2024-03-29T10:47:48.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.375+0000 7f438fd32640 10 monclient: init 2024-03-29T10:47:48.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.375+0000 7f438fd32640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:48.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.375+0000 7f438fd32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:48.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f438fd32640 10 monclient(hunting): picked mon.noname-a con 0x7f4388072450 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:48.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f438fd32640 10 monclient(hunting): picked mon.noname-b con 0x7f43880b0860 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:48.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f438fd32640 10 monclient(hunting): picked mon.noname-c con 0x7f438813ee40 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:48.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f438fd32640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43881503e0 con 0x7f4388072450 2024-03-29T10:47:48.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f438fd32640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438813cbb0 con 0x7f43880b0860 2024-03-29T10:47:48.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f438fd32640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43880b43c0 con 0x7f438813ee40 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f438fd32640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f438fd32640 10 monclient(hunting): authenticate will time out at 1088.573164s 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f438e2cf640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f438813ee40 0x7f43881e6710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:45896/0 (socket says 172.21.15.98:45896) 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f438e2cf640 1 -- 172.21.15.98:0/198865636 learned_addr learned my addr 172.21.15.98:0/198865636 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f437e7fc640 10 client.?.objecter ms_handle_connect 0x7f438813ee40 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f437e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f437e7fc640 10 client.?.objecter ms_handle_connect 0x7f4388072450 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f437e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175339650 0 0) 0x7f43880b43c0 con 0x7f438813ee40 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f437e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f437e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f437e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f437e7fc640 10 monclient(hunting): my global_id is 4337 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.376+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f436c004eb0 con 0x7f438813ee40 2024-03-29T10:47:48.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348142682 0 0) 0x7f43881503e0 con 0x7f4388072450 2024-03-29T10:47:48.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:48.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:48.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:48.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient(hunting): my global_id is 4329 2024-03-29T10:47:48.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43880b43c0 con 0x7f4388072450 2024-03-29T10:47:48.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 706001717 0 0) 0x7f436c004eb0 con 0x7f438813ee40 2024-03-29T10:47:48.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43881503e0 con 0x7f438813ee40 2024-03-29T10:47:48.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4380001b90 con 0x7f438813ee40 2024-03-29T10:47:48.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:48.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient(hunting): dump: 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3961146522 0 0) 0x7f43881503e0 con 0x7f438813ee40 2024-03-29T10:47:48.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 >> v1:172.21.15.189:6789/0 conn(0x7f43880b0860 legacy=0x7f438813d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 >> v1:172.21.15.98:6789/0 conn(0x7f4388072450 legacy=0x7f43880b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:48.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 1 monclient: found mon.c 2024-03-29T10:47:48.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:48.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:47:48.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43880b1830 con 0x7f438813ee40 2024-03-29T10:47:48.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:48.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:48.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:48.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f438fd32640 5 monclient: authenticate success, global_id 4337 2024-03-29T10:47:48.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f438fd32640 10 monclient: _renew_subs 2024-03-29T10:47:48.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f438fd32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:47:48.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f438fd32640 1 -- 172.21.15.98:0/198865636 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4388150680 con 0x7f438813ee40 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f438fd32640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f438fd32640 10 monclient: _renew_subs 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f438fd32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f438fd32640 1 -- 172.21.15.98:0/198865636 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43880b2890 con 0x7f438813ee40 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f438fd32640 1 librados: init done 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43800020a0 con 0x7f438813ee40 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4380004d00 con 0x7f438813ee40 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437ffff640 4 set_mon_vals no callback set 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.377+0000 7f437e7fc640 10 monclient: dump: 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:48.376 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.379+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f4380004e60 con 0x7f438813ee40 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.380+0000 7f438fd32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.380+0000 7f438fd32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.380+0000 7f438fd32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.380+0000 7f438fd32640 1 -- 172.21.15.98:0/198865636 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4370005080 con 0x7f438813ee40 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.380+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f4380084da0 con 0x7f438813ee40 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.380+0000 7f437e7fc640 10 client.4337.objecter ms_dispatch 0x7f4388000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.380+0000 7f437e7fc640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:48.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.380+0000 7f437e7fc640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:48.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.380+0000 7f437e7fc640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T10:47:48.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.381+0000 7f438dace640 10 monclient: get_auth_request con 0x7f436c06dcc0 auth_method 0 2024-03-29T10:47:48.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.381+0000 7f437e7fc640 10 client.4337.objecter ms_handle_connect 0x7f436c06dcc0 2024-03-29T10:47:48.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.388+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4380050680 con 0x7f438813ee40 2024-03-29T10:47:48.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.388+0000 7f437e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:48.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.388+0000 7f437e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:48.682 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.700+0000 7f438fd32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:48.682 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.700+0000 7f438fd32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:48.682 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.700+0000 7f438fd32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:47:48.682 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.700+0000 7f438fd32640 1 -- 172.21.15.98:0/198865636 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4370005330 con 0x7f438813ee40 2024-03-29T10:47:48.684 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.702+0000 7f437e7fc640 1 -- 172.21.15.98:0/198865636 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f43800525b0 con 0x7f438813ee40 2024-03-29T10:47:48.684 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.702+0000 7f437e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:48.684 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.702+0000 7f437e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:48.684 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:47:48.684 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:48.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.705+0000 7f438fd32640 10 librados: watch_flush enter 2024-03-29T10:47:48.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.705+0000 7f438fd32640 10 librados: watch_flush exit 2024-03-29T10:47:48.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.705+0000 7f438fd32640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T10:47:48.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.705+0000 7f438fd32640 10 client.4337.objecter successfully canceled tick 2024-03-29T10:47:48.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.705+0000 7f438fd32640 1 -- 172.21.15.98:0/198865636 >> v1:172.21.15.98:6816/1481799426 conn(0x7f436c06dcc0 legacy=0x7f436c070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.705+0000 7f438fd32640 10 monclient: shutdown 2024-03-29T10:47:48.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.705+0000 7f438fd32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:48.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.705+0000 7f438fd32640 1 -- 172.21.15.98:0/198865636 >> v1:172.21.15.98:6790/0 conn(0x7f438813ee40 legacy=0x7f43881e6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.705+0000 7f438fd32640 1 -- 172.21.15.98:0/198865636 shutdown_connections 2024-03-29T10:47:48.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.705+0000 7f438fd32640 1 -- 172.21.15.98:0/198865636 >> 172.21.15.98:0/198865636 conn(0x7f438806b660 msgr2=0x7f438806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:48.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.705+0000 7f438fd32640 1 -- 172.21.15.98:0/198865636 shutdown_connections 2024-03-29T10:47:48.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.705+0000 7f438fd32640 1 -- 172.21.15.98:0/198865636 wait complete. 2024-03-29T10:47:48.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:48.705+0000 7f438fd32640 1 librados: shutdown 2024-03-29T10:47:48.701 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:55.508 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:55.668 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.686+0000 7ff68121e640 10 monclient: get_monmap_and_config 2024-03-29T10:47:55.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.686+0000 7ff68121e640 10 monclient: build_initial_monmap 2024-03-29T10:47:55.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.686+0000 7ff68121e640 10 monclient: monmap: 2024-03-29T10:47:55.669 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:55.669 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:55.669 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:47:55.687598+0000 2024-03-29T10:47:55.669 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:47:55.687598+0000 2024-03-29T10:47:55.669 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:55.669 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:55.669 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:55.669 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:47:55.669 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:47:55.669 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:55.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.687+0000 7ff68121e640 1 Processor -- start 2024-03-29T10:47:55.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.687+0000 7ff68121e640 1 -- start start 2024-03-29T10:47:55.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.687+0000 7ff68121e640 10 monclient: init 2024-03-29T10:47:55.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.688+0000 7ff68121e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:55.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.689+0000 7ff68121e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:55.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.689+0000 7ff68121e640 10 monclient(hunting): picked mon.noname-c con 0x7ff67c1422b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:55.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.689+0000 7ff68121e640 10 monclient(hunting): picked mon.noname-b con 0x7ff67c142690 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:55.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.689+0000 7ff68121e640 10 monclient(hunting): picked mon.noname-a con 0x7ff67c0b17d0 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:55.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.689+0000 7ff68121e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c142c80 con 0x7ff67c0b17d0 2024-03-29T10:47:55.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.689+0000 7ff68121e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c0aae40 con 0x7ff67c1422b0 2024-03-29T10:47:55.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.689+0000 7ff68121e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c13fbd0 con 0x7ff67c142690 2024-03-29T10:47:55.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.689+0000 7ff68121e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:55.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.689+0000 7ff68121e640 10 monclient(hunting): authenticate will time out at 1095.886209s 2024-03-29T10:47:55.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.689+0000 7ff67ad9d640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff67c1422b0 0x7ff67c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:40632/0 (socket says 172.21.15.98:40632) 2024-03-29T10:47:55.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.689+0000 7ff67b59e640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff67c0b17d0 0x7ff67c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41502/0 (socket says 172.21.15.98:41502) 2024-03-29T10:47:55.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.689+0000 7ff67ad9d640 1 -- 172.21.15.98:0/2975552662 learned_addr learned my addr 172.21.15.98:0/2975552662 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:55.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 1 -- 172.21.15.98:0/2975552662 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962610505 0 0) 0x7ff67c142c80 con 0x7ff67c0b17d0 2024-03-29T10:47:55.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:47:55.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:55.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:55.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:47:55.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 1 -- 172.21.15.98:0/2975552662 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff664003520 con 0x7ff67c0b17d0 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 1 -- 172.21.15.98:0/2975552662 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785532062 0 0) 0x7ff67c0aae40 con 0x7ff67c1422b0 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 10 monclient(hunting): my global_id is 4342 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 1 -- 172.21.15.98:0/2975552662 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67c142c80 con 0x7ff67c1422b0 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 1 -- 172.21.15.98:0/2975552662 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 287337826 0 0) 0x7ff664003520 con 0x7ff67c0b17d0 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 1 -- 172.21.15.98:0/2975552662 >> v1:172.21.15.98:6790/0 conn(0x7ff67c142690 legacy=0x7ff67c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 1 -- 172.21.15.98:0/2975552662 >> v1:172.21.15.189:6789/0 conn(0x7ff67c1422b0 legacy=0x7ff67c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.690+0000 7ff679d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff679d9b640 1 monclient: found mon.noname-a 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff679d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff679d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:47:55.677 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff679d9b640 1 -- 172.21.15.98:0/2975552662 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c151940 con 0x7ff67c0b17d0 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff679d9b640 10 monclient: _finish_auth 0 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff679d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff679d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff679d9b640 1 -- 172.21.15.98:0/2975552662 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff66c001940 con 0x7ff67c0b17d0 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff679d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff679d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff679d9b640 10 monclient: dump: 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:55.678 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff68121e640 5 monclient: authenticate success, global_id 4332 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff68121e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.691+0000 7ff679d9b640 1 -- 172.21.15.98:0/2975552662 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff66c001b60 con 0x7ff67c0b17d0 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.692+0000 7ff679d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.692+0000 7ff68121e640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.692+0000 7ff68121e640 4 set_mon_vals no callback set 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.692+0000 7ff679d9b640 1 -- 172.21.15.98:0/2975552662 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff66c001e70 con 0x7ff67c0b17d0 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.692+0000 7ff679d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.692+0000 7ff679d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.692+0000 7ff679d9b640 10 monclient: dump: 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:55.679 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.693+0000 7ff68121e640 10 monclient: shutdown 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.693+0000 7ff68121e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.693+0000 7ff68121e640 1 -- 172.21.15.98:0/2975552662 >> v1:172.21.15.98:6789/0 conn(0x7ff67c0b17d0 legacy=0x7ff67c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.693+0000 7ff68121e640 1 -- 172.21.15.98:0/2975552662 shutdown_connections 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.693+0000 7ff68121e640 1 -- 172.21.15.98:0/2975552662 >> 172.21.15.98:0/2975552662 conn(0x7ff67c066a00 msgr2=0x7ff67c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.694+0000 7ff68121e640 1 -- 172.21.15.98:0/2975552662 shutdown_connections 2024-03-29T10:47:55.680 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.694+0000 7ff68121e640 1 -- 172.21.15.98:0/2975552662 wait complete. 2024-03-29T10:47:55.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.695+0000 7ff68121e640 10 monclient: build_initial_monmap 2024-03-29T10:47:55.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.695+0000 7ff68121e640 10 monclient: monmap: 2024-03-29T10:47:55.688 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:47:55.688 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:55.688 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:47:55.688 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:47:55.688 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:55.688 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:55.688 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:47:55.688 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:47:55.688 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.695+0000 7ff68121e640 1 librados: starting msgr at 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.695+0000 7ff68121e640 1 librados: starting objecter 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.696+0000 7ff68121e640 1 Processor -- start 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.696+0000 7ff68121e640 1 -- start start 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.696+0000 7ff68121e640 1 librados: setting wanted keys 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.696+0000 7ff68121e640 1 librados: calling monclient init 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.696+0000 7ff68121e640 10 monclient: init 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.697+0000 7ff68121e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.697+0000 7ff68121e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.697+0000 7ff68121e640 10 monclient(hunting): picked mon.noname-c con 0x7ff67c0b17d0 addr v1:172.21.15.98:6790/0 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.697+0000 7ff68121e640 10 monclient(hunting): picked mon.noname-b con 0x7ff67c1422b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.697+0000 7ff67ad9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7ff67c0b17d0 0x7ff67c071870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:42090/0 (socket says 172.21.15.98:42090) 2024-03-29T10:47:55.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.697+0000 7ff67ad9d640 1 -- 172.21.15.98:0/2253622590 learned_addr learned my addr 172.21.15.98:0/2253622590 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.697+0000 7ff68121e640 10 monclient(hunting): picked mon.noname-a con 0x7ff67c142690 addr v1:172.21.15.98:6789/0 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.697+0000 7ff68121e640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c070b90 con 0x7ff67c142690 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.697+0000 7ff6637fe640 10 client.?.objecter ms_handle_connect 0x7ff67c0b17d0 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.697+0000 7ff6637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.697+0000 7ff6637fe640 10 client.?.objecter ms_handle_connect 0x7ff67c1422b0 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.697+0000 7ff6637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff68121e640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c13c9d0 con 0x7ff67c1422b0 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff68121e640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c070e00 con 0x7ff67c0b17d0 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff68121e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff68121e640 10 monclient(hunting): authenticate will time out at 1095.895446s 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 client.?.objecter ms_handle_connect 0x7ff67c142690 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606800822 0 0) 0x7ff67c13c9d0 con 0x7ff67c1422b0 2024-03-29T10:47:55.690 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 monclient(hunting): my global_id is 4345 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff668004eb0 con 0x7ff67c1422b0 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631572377 0 0) 0x7ff67c070b90 con 0x7ff67c142690 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67c13c9d0 con 0x7ff67c142690 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386689921 0 0) 0x7ff67c070e00 con 0x7ff67c0b17d0 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:55.691 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.698+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67c070b90 con 0x7ff67c0b17d0 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2705784288 0 0) 0x7ff668004eb0 con 0x7ff67c1422b0 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff67c070e00 con 0x7ff67c1422b0 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff670001810 con 0x7ff67c1422b0 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient(hunting): dump: 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:55.692 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 86413518 0 0) 0x7ff67c070b90 con 0x7ff67c0b17d0 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff668004eb0 con 0x7ff67c0b17d0 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff6740019b0 con 0x7ff67c0b17d0 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient(hunting): dump: 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:55.693 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3986136173 0 0) 0x7ff67c13c9d0 con 0x7ff67c142690 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff67c070b90 con 0x7ff67c142690 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff66c001660 con 0x7ff67c142690 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient(hunting): dump: 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:55.694 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:55.695 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:55.695 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:55.695 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:55.695 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:55.695 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:55.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1431975989 0 0) 0x7ff67c070e00 con 0x7ff67c1422b0 2024-03-29T10:47:55.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 >> v1:172.21.15.98:6790/0 conn(0x7ff67c0b17d0 legacy=0x7ff67c071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:55.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 >> v1:172.21.15.98:6789/0 conn(0x7ff67c142690 legacy=0x7ff67c06eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:55.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:55.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 monclient: found mon.b 2024-03-29T10:47:55.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c0b0eb0 con 0x7ff67c1422b0 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff6637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff68121e640 5 monclient: authenticate success, global_id 4345 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff68121e640 10 monclient: _renew_subs 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff68121e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.699+0000 7ff68121e640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff67c13c420 con 0x7ff67c1422b0 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.700+0000 7ff68121e640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.700+0000 7ff68121e640 10 monclient: _renew_subs 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.700+0000 7ff68121e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.700+0000 7ff68121e640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67c06f5e0 con 0x7ff67c1422b0 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.700+0000 7ff68121e640 1 librados: init done 2024-03-29T10:47:55.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.700+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff670001d00 con 0x7ff67c1422b0 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.700+0000 7ff6637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.700+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff670003790 con 0x7ff67c1422b0 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.700+0000 7ff6637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.700+0000 7ff6637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.700+0000 7ff6637fe640 10 monclient: dump: 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:47:55.697 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.700+0000 7ff67959a640 4 set_mon_vals no callback set 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.701+0000 7ff68121e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.701+0000 7ff68121e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.701+0000 7ff68121e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.701+0000 7ff68121e640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff65c005080 con 0x7ff67c1422b0 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.702+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7ff6700038f0 con 0x7ff67c1422b0 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.703+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7ff670083750 con 0x7ff67c1422b0 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.703+0000 7ff6637fe640 10 client.4345.objecter ms_dispatch 0x7ff67c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.703+0000 7ff6637fe640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.703+0000 7ff6637fe640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.703+0000 7ff6637fe640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.703+0000 7ff67ad9d640 10 monclient: get_auth_request con 0x7ff66806e120 auth_method 0 2024-03-29T10:47:55.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.704+0000 7ff6637fe640 10 client.4345.objecter ms_handle_connect 0x7ff66806e120 2024-03-29T10:47:55.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.710+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff67004f030 con 0x7ff67c1422b0 2024-03-29T10:47:55.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.710+0000 7ff6637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:55.699 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:55.710+0000 7ff6637fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:56.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.114+0000 7ff68121e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:56.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.114+0000 7ff68121e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:56.097 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.114+0000 7ff68121e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:47:56.097 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.114+0000 7ff68121e640 1 -- 172.21.15.98:0/2253622590 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff65c005330 con 0x7ff67c1422b0 2024-03-29T10:47:56.097 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.115+0000 7ff6637fe640 1 -- 172.21.15.98:0/2253622590 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7ff670050f60 con 0x7ff67c1422b0 2024-03-29T10:47:56.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.116+0000 7ff6637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:56.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.116+0000 7ff6637fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:56.098 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:47:56.098 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:56.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.120+0000 7ff68121e640 10 librados: watch_flush enter 2024-03-29T10:47:56.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.120+0000 7ff68121e640 10 librados: watch_flush exit 2024-03-29T10:47:56.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.120+0000 7ff68121e640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-29T10:47:56.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.120+0000 7ff68121e640 10 client.4345.objecter successfully canceled tick 2024-03-29T10:47:56.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.120+0000 7ff68121e640 1 -- 172.21.15.98:0/2253622590 >> v1:172.21.15.98:6816/1481799426 conn(0x7ff66806e120 legacy=0x7ff6680705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:56.102 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.120+0000 7ff68121e640 10 monclient: shutdown 2024-03-29T10:47:56.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.120+0000 7ff68121e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:56.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.120+0000 7ff68121e640 1 -- 172.21.15.98:0/2253622590 >> v1:172.21.15.189:6789/0 conn(0x7ff67c1422b0 legacy=0x7ff67c13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:56.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.120+0000 7ff68121e640 1 -- 172.21.15.98:0/2253622590 shutdown_connections 2024-03-29T10:47:56.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.120+0000 7ff68121e640 1 -- 172.21.15.98:0/2253622590 >> 172.21.15.98:0/2253622590 conn(0x7ff67c066a00 msgr2=0x7ff67c13ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:56.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.120+0000 7ff68121e640 1 -- 172.21.15.98:0/2253622590 shutdown_connections 2024-03-29T10:47:56.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.120+0000 7ff68121e640 1 -- 172.21.15.98:0/2253622590 wait complete. 2024-03-29T10:47:56.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:47:56.120+0000 7ff68121e640 1 librados: shutdown 2024-03-29T10:47:56.118 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:02.924 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:03.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.083+0000 7f5a2208f640 10 monclient: get_monmap_and_config 2024-03-29T10:48:03.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.083+0000 7f5a2208f640 10 monclient: build_initial_monmap 2024-03-29T10:48:03.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.083+0000 7f5a2208f640 10 monclient: monmap: 2024-03-29T10:48:03.066 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:03.066 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:03.066 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:48:03.084803+0000 2024-03-29T10:48:03.066 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:48:03.084803+0000 2024-03-29T10:48:03.066 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:03.066 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:03.066 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:03.067 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:48:03.067 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:48:03.067 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:03.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.085+0000 7f5a2208f640 1 Processor -- start 2024-03-29T10:48:03.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.085+0000 7f5a2208f640 1 -- start start 2024-03-29T10:48:03.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.085+0000 7f5a2208f640 10 monclient: init 2024-03-29T10:48:03.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a2208f640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:03.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a2208f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:03.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a2208f640 10 monclient(hunting): picked mon.noname-a con 0x7f5a1c0708a0 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:03.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a2208f640 10 monclient(hunting): picked mon.noname-c con 0x7f5a1c06fe90 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:03.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a2208f640 10 monclient(hunting): picked mon.noname-b con 0x7f5a1c143c50 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:03.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a2208f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1c072630 con 0x7f5a1c0708a0 2024-03-29T10:48:03.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a2208f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1c071410 con 0x7f5a1c06fe90 2024-03-29T10:48:03.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a2208f640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1c0afaa0 con 0x7f5a1c143c50 2024-03-29T10:48:03.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a2208f640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a2208f640 10 monclient(hunting): authenticate will time out at 1103.283351s 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a1b7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5a1c0708a0 0x7f5a1c0af390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34204/0 (socket says 172.21.15.98:34204) 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a1b7fe640 1 -- 172.21.15.98:0/22801837 learned_addr learned my addr 172.21.15.98:0/22801837 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a1bfff640 1 --1- 172.21.15.98:0/22801837 >> v1:172.21.15.98:6790/0 conn(0x7f5a1c143c50 0x7f5a1c14b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:55998/0 (socket says 172.21.15.98:55998) 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a1affd640 1 -- 172.21.15.98:0/22801837 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964302672 0 0) 0x7f5a1c0afaa0 con 0x7f5a1c143c50 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a1affd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a1affd640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.086+0000 7f5a1affd640 1 -- 172.21.15.98:0/22801837 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a08007930 con 0x7f5a1c143c50 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 1 -- 172.21.15.98:0/22801837 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2436759077 0 0) 0x7f5a08007930 con 0x7f5a1c143c50 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 1 -- 172.21.15.98:0/22801837 >> v1:172.21.15.189:6789/0 conn(0x7f5a1c06fe90 legacy=0x7f5a1c143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 1 -- 172.21.15.98:0/22801837 >> v1:172.21.15.98:6789/0 conn(0x7f5a1c0708a0 legacy=0x7f5a1c0af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:03.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 1 monclient: found mon.noname-b 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 1 -- 172.21.15.98:0/22801837 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c0ac8b0 con 0x7f5a1c143c50 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 10 monclient: _finish_auth 0 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a2208f640 5 monclient: authenticate success, global_id 4343 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 1 -- 172.21.15.98:0/22801837 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5a14001940 con 0x7f5a1c143c50 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a2208f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a1affd640 10 monclient: dump: 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:03.071 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:03.072 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:03.072 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:03.072 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:03.072 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:03.072 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:03.072 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:03.072 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:03.072 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:03.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.087+0000 7f5a2208f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:03.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.088+0000 7f5a1affd640 1 -- 172.21.15.98:0/22801837 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a14001b60 con 0x7f5a1c143c50 2024-03-29T10:48:03.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.088+0000 7f5a1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:03.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.088+0000 7f5a1affd640 1 -- 172.21.15.98:0/22801837 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5a14001ef0 con 0x7f5a1c143c50 2024-03-29T10:48:03.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.088+0000 7f5a1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:03.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.088+0000 7f5a1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:03.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.088+0000 7f5a1affd640 10 monclient: dump: 2024-03-29T10:48:03.076 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:03.076 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:03.076 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:03.076 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:03.076 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:03.076 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:03.076 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.088+0000 7f5a2208f640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.088+0000 7f5a2208f640 4 set_mon_vals no callback set 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.089+0000 7f5a2208f640 10 monclient: shutdown 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.089+0000 7f5a2208f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.089+0000 7f5a2208f640 1 -- 172.21.15.98:0/22801837 >> v1:172.21.15.98:6790/0 conn(0x7f5a1c143c50 legacy=0x7f5a1c14b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.089+0000 7f5a2208f640 1 -- 172.21.15.98:0/22801837 shutdown_connections 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.089+0000 7f5a2208f640 1 -- 172.21.15.98:0/22801837 >> 172.21.15.98:0/22801837 conn(0x7f5a1c0675b0 msgr2=0x7f5a1c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.089+0000 7f5a2208f640 1 -- 172.21.15.98:0/22801837 shutdown_connections 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.089+0000 7f5a2208f640 1 -- 172.21.15.98:0/22801837 wait complete. 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.090+0000 7f5a2208f640 10 monclient: build_initial_monmap 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.090+0000 7f5a2208f640 10 monclient: monmap: 2024-03-29T10:48:03.077 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.090+0000 7f5a2208f640 1 librados: starting msgr at 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.090+0000 7f5a2208f640 1 librados: starting objecter 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.090+0000 7f5a2208f640 1 Processor -- start 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.090+0000 7f5a2208f640 1 -- start start 2024-03-29T10:48:03.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.090+0000 7f5a2208f640 1 librados: setting wanted keys 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.090+0000 7f5a2208f640 1 librados: calling monclient init 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.090+0000 7f5a2208f640 10 monclient: init 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.091+0000 7f5a2208f640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.091+0000 7f5a2208f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.091+0000 7f5a2208f640 10 monclient(hunting): picked mon.noname-c con 0x7f5a1c06fe90 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.091+0000 7f5a2208f640 10 monclient(hunting): picked mon.noname-b con 0x7f5a1c0708a0 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.091+0000 7f5a2208f640 10 monclient(hunting): picked mon.noname-a con 0x7f5a1c143c50 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.091+0000 7f5a2208f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1c140b90 con 0x7f5a1c143c50 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.091+0000 7f5a2208f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1c140870 con 0x7f5a1c0708a0 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.091+0000 7f5a2208f640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1c13f980 con 0x7f5a1c06fe90 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.091+0000 7f5a2208f640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.091+0000 7f5a2208f640 10 monclient(hunting): authenticate will time out at 1103.288795s 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a13fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5a1c0708a0 0x7f5a1c0abcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:57290/0 (socket says 172.21.15.98:57290) 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a13fff640 1 -- 172.21.15.98:0/2520384197 learned_addr learned my addr 172.21.15.98:0/2520384197 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:03.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 10 client.?.objecter ms_handle_connect 0x7f5a1c0708a0 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 10 client.?.objecter ms_handle_connect 0x7f5a1c06fe90 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487471475 0 0) 0x7f5a1c140870 con 0x7f5a1c0708a0 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 10 monclient(hunting): my global_id is 4351 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59fc004eb0 con 0x7f5a1c0708a0 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496001876 0 0) 0x7f5a1c13f980 con 0x7f5a1c06fe90 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:48:03.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.092+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a1c140870 con 0x7f5a1c06fe90 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2710866774 0 0) 0x7f5a1c140870 con 0x7f5a1c06fe90 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a1c13f980 con 0x7f5a1c06fe90 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5a0c0018a0 con 0x7f5a1c06fe90 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 10 monclient(hunting): dump: 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:03.081 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 10 client.?.objecter ms_handle_connect 0x7f5a1c143c50 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2919485620 0 0) 0x7f59fc004eb0 con 0x7f5a1c0708a0 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a1c140870 con 0x7f5a1c0708a0 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 187414112 0 0) 0x7f5a1c13f980 con 0x7f5a1c06fe90 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 >> v1:172.21.15.189:6789/0 conn(0x7f5a1c0708a0 legacy=0x7f5a1c0abcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 >> v1:172.21.15.98:6789/0 conn(0x7f5a1c143c50 legacy=0x7f5a1c1d30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 1 monclient: found mon.c 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:03.082 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.093+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c0ac400 con 0x7f5a1c06fe90 2024-03-29T10:48:03.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a18ff9640 10 monclient: _finish_auth 0 2024-03-29T10:48:03.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a18ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:48:03.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:03.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a2208f640 5 monclient: authenticate success, global_id 4346 2024-03-29T10:48:03.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a2208f640 10 monclient: _renew_subs 2024-03-29T10:48:03.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a2208f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:03.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a2208f640 1 -- 172.21.15.98:0/2520384197 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1c0ac6b0 con 0x7f5a1c06fe90 2024-03-29T10:48:03.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a2208f640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:03.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a2208f640 10 monclient: _renew_subs 2024-03-29T10:48:03.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a2208f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:03.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a2208f640 1 -- 172.21.15.98:0/2520384197 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1c142dd0 con 0x7f5a1c06fe90 2024-03-29T10:48:03.083 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a2208f640 1 librados: init done 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a0c0047e0 con 0x7f5a1c06fe90 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a18ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5a0c004b30 con 0x7f5a1c06fe90 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a1a7fc640 4 set_mon_vals no callback set 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.094+0000 7f5a18ff9640 10 monclient: dump: 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:03.085 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.095+0000 7f5a2208f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.095+0000 7f5a2208f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.096+0000 7f5a2208f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.096+0000 7f5a2208f640 1 -- 172.21.15.98:0/2520384197 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a00005080 con 0x7f5a1c06fe90 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.097+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f5a0c01ab00 con 0x7f5a1c06fe90 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.097+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f5a0c0848a0 con 0x7f5a1c06fe90 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.097+0000 7f5a18ff9640 10 client.4346.objecter ms_dispatch 0x7f5a1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.097+0000 7f5a18ff9640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.097+0000 7f5a18ff9640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.097+0000 7f5a13fff640 10 monclient: get_auth_request con 0x7f59fc06ddf0 auth_method 0 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.097+0000 7f5a18ff9640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-29T10:48:03.086 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.098+0000 7f5a18ff9640 10 client.4346.objecter ms_handle_connect 0x7f59fc06ddf0 2024-03-29T10:48:03.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.106+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a0c050180 con 0x7f5a1c06fe90 2024-03-29T10:48:03.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.106+0000 7f5a18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:03.088 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.106+0000 7f5a18ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:03.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.470+0000 7f5a2208f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:03.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.470+0000 7f5a2208f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:03.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.470+0000 7f5a2208f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:03.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.470+0000 7f5a2208f640 1 -- 172.21.15.98:0/2520384197 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a00005330 con 0x7f5a1c06fe90 2024-03-29T10:48:03.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.471+0000 7f5a18ff9640 1 -- 172.21.15.98:0/2520384197 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f5a0c0520b0 con 0x7f5a1c06fe90 2024-03-29T10:48:03.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.471+0000 7f5a18ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:03.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.471+0000 7f5a18ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:03.454 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:48:03.454 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:03.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.475+0000 7f5a2208f640 10 librados: watch_flush enter 2024-03-29T10:48:03.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.475+0000 7f5a2208f640 10 librados: watch_flush exit 2024-03-29T10:48:03.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.475+0000 7f5a2208f640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-29T10:48:03.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.475+0000 7f5a2208f640 10 client.4346.objecter successfully canceled tick 2024-03-29T10:48:03.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.475+0000 7f5a2208f640 1 -- 172.21.15.98:0/2520384197 >> v1:172.21.15.98:6816/1481799426 conn(0x7f59fc06ddf0 legacy=0x7f59fc0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:03.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.475+0000 7f5a2208f640 10 monclient: shutdown 2024-03-29T10:48:03.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.475+0000 7f5a2208f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:03.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.475+0000 7f5a2208f640 1 -- 172.21.15.98:0/2520384197 >> v1:172.21.15.98:6790/0 conn(0x7f5a1c06fe90 legacy=0x7f5a1c140160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:03.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.476+0000 7f5a2208f640 1 -- 172.21.15.98:0/2520384197 shutdown_connections 2024-03-29T10:48:03.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.476+0000 7f5a2208f640 1 -- 172.21.15.98:0/2520384197 >> 172.21.15.98:0/2520384197 conn(0x7f5a1c0675b0 msgr2=0x7f5a1c0af300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:03.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.476+0000 7f5a2208f640 1 -- 172.21.15.98:0/2520384197 shutdown_connections 2024-03-29T10:48:03.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.476+0000 7f5a2208f640 1 -- 172.21.15.98:0/2520384197 wait complete. 2024-03-29T10:48:03.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:03.476+0000 7f5a2208f640 1 librados: shutdown 2024-03-29T10:48:03.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:10.280 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:10.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.448+0000 7f7ffe064640 10 monclient: get_monmap_and_config 2024-03-29T10:48:10.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.448+0000 7f7ffe064640 10 monclient: build_initial_monmap 2024-03-29T10:48:10.433 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.448+0000 7f7ffe064640 10 monclient: monmap: 2024-03-29T10:48:10.433 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:10.433 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:10.434 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:48:10.449370+0000 2024-03-29T10:48:10.434 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:48:10.449370+0000 2024-03-29T10:48:10.434 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:10.434 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:10.434 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:10.434 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:48:10.434 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:48:10.434 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:10.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.451+0000 7f7ffe064640 1 Processor -- start 2024-03-29T10:48:10.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.451+0000 7f7ffe064640 1 -- start start 2024-03-29T10:48:10.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.451+0000 7f7ffe064640 10 monclient: init 2024-03-29T10:48:10.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.452+0000 7f7ffe064640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:10.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.452+0000 7f7ffe064640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:10.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.452+0000 7f7ffe064640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff80af0f0 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:10.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.452+0000 7f7ff77fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7ff80af0f0 0x7f7ff813c3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41704/0 (socket says 172.21.15.98:41704) 2024-03-29T10:48:10.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.453+0000 7f7ffe064640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff80af4d0 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:10.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.453+0000 7f7ffe064640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff80ab170 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:10.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.453+0000 7f7ffe064640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff813f010 con 0x7f7ff80af0f0 2024-03-29T10:48:10.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.453+0000 7f7ffe064640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80ab6c0 con 0x7f7ff80ab170 2024-03-29T10:48:10.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.453+0000 7f7ffe064640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80ab9a0 con 0x7f7ff80af4d0 2024-03-29T10:48:10.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.453+0000 7f7ffe064640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:10.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.453+0000 7f7ffe064640 10 monclient(hunting): authenticate will time out at 1110.650077s 2024-03-29T10:48:10.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.453+0000 7f7ff77fe640 1 -- 172.21.15.98:0/3679427543 learned_addr learned my addr 172.21.15.98:0/3679427543 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:10.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.455+0000 7f7ff67fc640 1 -- 172.21.15.98:0/3679427543 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393381891 0 0) 0x7f7ff80ab9a0 con 0x7f7ff80af4d0 2024-03-29T10:48:10.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.455+0000 7f7ff67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:10.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.455+0000 7f7ff67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:10.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.455+0000 7f7ff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:10.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.455+0000 7f7ff67fc640 10 monclient(hunting): my global_id is 4349 2024-03-29T10:48:10.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.455+0000 7f7ff67fc640 1 -- 172.21.15.98:0/3679427543 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe0003520 con 0x7f7ff80af4d0 2024-03-29T10:48:10.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.455+0000 7f7ff67fc640 1 -- 172.21.15.98:0/3679427543 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1305624052 0 0) 0x7f7fe0003520 con 0x7f7ff80af4d0 2024-03-29T10:48:10.438 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.456+0000 7f7ff67fc640 1 -- 172.21.15.98:0/3679427543 >> v1:172.21.15.189:6789/0 conn(0x7f7ff80ab170 legacy=0x7f7ff814a850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:10.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.457+0000 7f7ff67fc640 1 -- 172.21.15.98:0/3679427543 >> v1:172.21.15.98:6789/0 conn(0x7f7ff80af0f0 legacy=0x7f7ff813c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:10.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.457+0000 7f7ff67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:10.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.457+0000 7f7ff67fc640 1 monclient: found mon.noname-b 2024-03-29T10:48:10.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.457+0000 7f7ff67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:10.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:48:10.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 1 -- 172.21.15.98:0/3679427543 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff80abc80 con 0x7f7ff80af4d0 2024-03-29T10:48:10.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:10.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:10.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:10.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 1 -- 172.21.15.98:0/3679427543 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f7ff0001940 con 0x7f7ff80af4d0 2024-03-29T10:48:10.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 10 monclient: dump: 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 1 -- 172.21.15.98:0/3679427543 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ff0001bd0 con 0x7f7ff80af4d0 2024-03-29T10:48:10.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 1 -- 172.21.15.98:0/3679427543 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f7ff0001f60 con 0x7f7ff80af4d0 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.458+0000 7f7ff67fc640 10 monclient: dump: 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:10.443 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:10.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.459+0000 7f7ffe064640 5 monclient: authenticate success, global_id 4349 2024-03-29T10:48:10.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.459+0000 7f7ffe064640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:10.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.459+0000 7f7ffe064640 4 set_mon_vals no callback set 2024-03-29T10:48:10.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.460+0000 7f7ffe064640 10 monclient: shutdown 2024-03-29T10:48:10.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.460+0000 7f7ffe064640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:10.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.460+0000 7f7ffe064640 1 -- 172.21.15.98:0/3679427543 >> v1:172.21.15.98:6790/0 conn(0x7f7ff80af4d0 legacy=0x7f7ff813f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:10.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.460+0000 7f7ffe064640 1 -- 172.21.15.98:0/3679427543 shutdown_connections 2024-03-29T10:48:10.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.460+0000 7f7ffe064640 1 -- 172.21.15.98:0/3679427543 >> 172.21.15.98:0/3679427543 conn(0x7f7ff80b35e0 msgr2=0x7f7ff80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:10.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.461+0000 7f7ffe064640 1 -- 172.21.15.98:0/3679427543 shutdown_connections 2024-03-29T10:48:10.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.464+0000 7f7ffe064640 1 -- 172.21.15.98:0/3679427543 wait complete. 2024-03-29T10:48:10.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.465+0000 7f7ffe064640 10 monclient: build_initial_monmap 2024-03-29T10:48:10.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.465+0000 7f7ffe064640 10 monclient: monmap: 2024-03-29T10:48:10.447 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:10.447 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:10.447 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:48:10.447 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:48:10.447 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:10.447 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:10.447 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:10.447 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:48:10.447 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:48:10.447 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:10.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.465+0000 7f7ffe064640 1 librados: starting msgr at 2024-03-29T10:48:10.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.466+0000 7f7ffe064640 1 librados: starting objecter 2024-03-29T10:48:10.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.466+0000 7f7ffe064640 1 Processor -- start 2024-03-29T10:48:10.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.466+0000 7f7ffe064640 1 -- start start 2024-03-29T10:48:10.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.466+0000 7f7ffe064640 1 librados: setting wanted keys 2024-03-29T10:48:10.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.466+0000 7f7ffe064640 1 librados: calling monclient init 2024-03-29T10:48:10.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.466+0000 7f7ffe064640 10 monclient: init 2024-03-29T10:48:10.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.467+0000 7f7ffe064640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:10.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.467+0000 7f7ffe064640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:10.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.467+0000 7f7ffe064640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff80ab170 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:10.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.472+0000 7f7ffe064640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff80af0f0 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:10.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.472+0000 7f7ff77fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f7ff80ab170 0x7f7ff806fb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:51255/0 (socket says 172.21.15.98:51255) 2024-03-29T10:48:10.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.472+0000 7f7ff77fe640 1 -- 172.21.15.98:0/1691604324 learned_addr learned my addr 172.21.15.98:0/1691604324 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.472+0000 7f7fdffff640 10 client.?.objecter ms_handle_connect 0x7f7ff80ab170 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.472+0000 7f7ffe064640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff8131dc0 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.472+0000 7f7fdffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.472+0000 7f7fdffff640 10 client.?.objecter ms_handle_connect 0x7f7ff8131dc0 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.472+0000 7f7fdffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.472+0000 7f7ffe064640 1 -- 172.21.15.98:0/1691604324 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8071a70 con 0x7f7ff8131dc0 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.473+0000 7f7ffe064640 1 -- 172.21.15.98:0/1691604324 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8070e20 con 0x7f7ff80af0f0 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.473+0000 7f7ffe064640 1 -- 172.21.15.98:0/1691604324 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8071090 con 0x7f7ff80ab170 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.473+0000 7f7ffe064640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.473+0000 7f7ffe064640 10 monclient(hunting): authenticate will time out at 1110.670228s 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.473+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020512882 0 0) 0x7f7ff8071090 con 0x7f7ff80ab170 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.473+0000 7f7fdffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.473+0000 7f7fdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:10.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.473+0000 7f7fdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:10.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.473+0000 7f7fdffff640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:48:10.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.473+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe4004eb0 con 0x7f7ff80ab170 2024-03-29T10:48:10.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178021512 0 0) 0x7f7ff8071a70 con 0x7f7ff8131dc0 2024-03-29T10:48:10.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:10.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:10.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:10.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 10 monclient(hunting): my global_id is 4347 2024-03-29T10:48:10.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff8071090 con 0x7f7ff8131dc0 2024-03-29T10:48:10.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 10 client.?.objecter ms_handle_connect 0x7f7ff80af0f0 2024-03-29T10:48:10.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1764537558 0 0) 0x7f7fe4004eb0 con 0x7f7ff80ab170 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ff8071a70 con 0x7f7ff80ab170 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f7fec001950 con 0x7f7ff80ab170 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 10 monclient(hunting): dump: 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:10.463 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.475+0000 7f7fdffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813122419 0 0) 0x7f7ff8070e20 con 0x7f7ff80af0f0 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 10 monclient(hunting): my global_id is 4354 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe4004eb0 con 0x7f7ff80af0f0 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3423453423 0 0) 0x7f7ff8071a70 con 0x7f7ff80ab170 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 >> v1:172.21.15.189:6789/0 conn(0x7f7ff80af0f0 legacy=0x7f7ff813e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 >> v1:172.21.15.98:6789/0 conn(0x7f7ff8131dc0 legacy=0x7f7ff81dff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 1 monclient: found mon.c 2024-03-29T10:48:10.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff8071370 con 0x7f7ff80ab170 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 10 monclient: _finish_auth 0 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.476+0000 7f7fdffff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7fdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fec0047e0 con 0x7f7ff80ab170 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7ffe064640 5 monclient: authenticate success, global_id 4352 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7ffe064640 10 monclient: _renew_subs 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7ffe064640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7ffe064640 1 -- 172.21.15.98:0/1691604324 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff8071650 con 0x7f7ff80ab170 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7ffe064640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7ffe064640 10 monclient: _renew_subs 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7ffe064640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7ffe064640 1 -- 172.21.15.98:0/1691604324 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff81e1650 con 0x7f7ff80ab170 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7ffe064640 1 librados: init done 2024-03-29T10:48:10.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7fdffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f7fec004af0 con 0x7f7ff80ab170 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7fdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7fdffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.477+0000 7f7fdffff640 10 monclient: dump: 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:10.466 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:10.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.478+0000 7f7ff57fa640 4 set_mon_vals no callback set 2024-03-29T10:48:10.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.478+0000 7f7fddffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:10.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.479+0000 7f7fddffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:10.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.479+0000 7f7fddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:10.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.479+0000 7f7fddffb640 1 -- 172.21.15.98:0/1691604324 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd8005080 con 0x7f7ff80ab170 2024-03-29T10:48:10.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.480+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f7fec01aac0 con 0x7f7ff80ab170 2024-03-29T10:48:10.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.484+0000 7f7ff6ffd640 10 monclient: get_auth_request con 0x7f7fe406dea0 auth_method 0 2024-03-29T10:48:10.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.484+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f7fec084a30 con 0x7f7ff80ab170 2024-03-29T10:48:10.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.484+0000 7f7fdffff640 10 client.4352.objecter ms_dispatch 0x7f7ff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:10.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.484+0000 7f7fdffff640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:10.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.484+0000 7f7fdffff640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:10.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.485+0000 7f7fdffff640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T10:48:10.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.485+0000 7f7fdffff640 10 client.4352.objecter ms_handle_connect 0x7f7fe406dea0 2024-03-29T10:48:10.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.489+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fec050310 con 0x7f7ff80ab170 2024-03-29T10:48:10.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.489+0000 7f7fdffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:10.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.489+0000 7f7fdffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:10.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.817+0000 7f7fddffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:10.800 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.817+0000 7f7fddffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:10.800 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.817+0000 7f7fddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:10.800 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.817+0000 7f7fddffb640 1 -- 172.21.15.98:0/1691604324 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fd8005330 con 0x7f7ff80ab170 2024-03-29T10:48:10.801 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.819+0000 7f7fdffff640 1 -- 172.21.15.98:0/1691604324 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f7fec052240 con 0x7f7ff80ab170 2024-03-29T10:48:10.802 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.820+0000 7f7fdffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:10.802 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.820+0000 7f7fdffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:10.802 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:48:10.802 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:10.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.823+0000 7f7fddffb640 10 librados: watch_flush enter 2024-03-29T10:48:10.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.823+0000 7f7fddffb640 10 librados: watch_flush exit 2024-03-29T10:48:10.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.823+0000 7f7fddffb640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T10:48:10.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.823+0000 7f7fddffb640 10 client.4352.objecter successfully canceled tick 2024-03-29T10:48:10.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.824+0000 7f7fddffb640 1 -- 172.21.15.98:0/1691604324 >> v1:172.21.15.98:6816/1481799426 conn(0x7f7fe406dea0 legacy=0x7f7fe4070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:10.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.824+0000 7f7fddffb640 10 monclient: shutdown 2024-03-29T10:48:10.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.824+0000 7f7fddffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:10.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.824+0000 7f7fddffb640 1 -- 172.21.15.98:0/1691604324 >> v1:172.21.15.98:6790/0 conn(0x7f7ff80ab170 legacy=0x7f7ff806fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:10.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.824+0000 7f7fddffb640 1 -- 172.21.15.98:0/1691604324 shutdown_connections 2024-03-29T10:48:10.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.824+0000 7f7fddffb640 1 -- 172.21.15.98:0/1691604324 >> 172.21.15.98:0/1691604324 conn(0x7f7ff80b35e0 msgr2=0x7f7ff80ae900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:10.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.824+0000 7f7fddffb640 1 -- 172.21.15.98:0/1691604324 shutdown_connections 2024-03-29T10:48:10.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.824+0000 7f7fddffb640 1 -- 172.21.15.98:0/1691604324 wait complete. 2024-03-29T10:48:10.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:10.824+0000 7f7fddffb640 1 librados: shutdown 2024-03-29T10:48:10.824 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:17.630 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:17.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.802+0000 7f8061962640 10 monclient: get_monmap_and_config 2024-03-29T10:48:17.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.802+0000 7f8061962640 10 monclient: build_initial_monmap 2024-03-29T10:48:17.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.802+0000 7f8061962640 10 monclient: monmap: 2024-03-29T10:48:17.784 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:17.784 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:17.784 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:48:17.803004+0000 2024-03-29T10:48:17.784 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:48:17.803004+0000 2024-03-29T10:48:17.785 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:17.785 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:17.785 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:17.785 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:48:17.785 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:48:17.785 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:17.786 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.804+0000 7f8061962640 1 Processor -- start 2024-03-29T10:48:17.787 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.805+0000 7f8061962640 1 -- start start 2024-03-29T10:48:17.787 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.805+0000 7f8061962640 10 monclient: init 2024-03-29T10:48:17.787 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.806+0000 7f8061962640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:17.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.806+0000 7f8061962640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:17.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.806+0000 7f8061962640 10 monclient(hunting): picked mon.noname-a con 0x7f8054006a70 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:17.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.806+0000 7f8061962640 10 monclient(hunting): picked mon.noname-b con 0x7f805402b100 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:17.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.806+0000 7f8061962640 10 monclient(hunting): picked mon.noname-c con 0x7f8054029c30 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:17.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.806+0000 7f8061962640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805402a270 con 0x7f8054006a70 2024-03-29T10:48:17.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.806+0000 7f8061962640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805402c440 con 0x7f8054029c30 2024-03-29T10:48:17.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.806+0000 7f8061962640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80540c9440 con 0x7f805402b100 2024-03-29T10:48:17.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.806+0000 7f8061962640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:17.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.806+0000 7f8061962640 10 monclient(hunting): authenticate will time out at 1118.003828s 2024-03-29T10:48:17.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.807+0000 7f805bfff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f805402b100 0x7f80540cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:45906/0 (socket says 172.21.15.98:45906) 2024-03-29T10:48:17.790 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.807+0000 7f805bfff640 1 -- 172.21.15.98:0/1798548414 learned_addr learned my addr 172.21.15.98:0/1798548414 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:17.790 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.808+0000 7f805b7fe640 1 -- 172.21.15.98:0/1798548414 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906691209 0 0) 0x7f80540c9440 con 0x7f805402b100 2024-03-29T10:48:17.790 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.808+0000 7f805b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:17.790 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.808+0000 7f805b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:17.790 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.808+0000 7f805b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:17.791 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.808+0000 7f805b7fe640 10 monclient(hunting): my global_id is 4355 2024-03-29T10:48:17.791 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.808+0000 7f805b7fe640 1 -- 172.21.15.98:0/1798548414 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f804c003520 con 0x7f805402b100 2024-03-29T10:48:17.791 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.808+0000 7f805b7fe640 1 -- 172.21.15.98:0/1798548414 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822959240 0 0) 0x7f804c003520 con 0x7f805402b100 2024-03-29T10:48:17.791 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.808+0000 7f805b7fe640 1 -- 172.21.15.98:0/1798548414 >> v1:172.21.15.189:6789/0 conn(0x7f8054029c30 legacy=0x7f80540d74a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:17.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 1 -- 172.21.15.98:0/1798548414 >> v1:172.21.15.98:6789/0 conn(0x7f8054006a70 legacy=0x7f80540c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:17.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:17.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 1 monclient: found mon.noname-b 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 1 -- 172.21.15.98:0/1798548414 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80540c95d0 con 0x7f805402b100 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 1 -- 172.21.15.98:0/1798548414 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8050001670 con 0x7f805402b100 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 10 monclient: dump: 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:17.796 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f8061962640 5 monclient: authenticate success, global_id 4355 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f8061962640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 1 -- 172.21.15.98:0/1798548414 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80500018d0 con 0x7f805402b100 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f805b7fe640 1 -- 172.21.15.98:0/1798548414 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8050001d20 con 0x7f805402b100 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f8061962640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.811+0000 7f8061962640 4 set_mon_vals no callback set 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.812+0000 7f805b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:17.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.812+0000 7f805b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:17.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.812+0000 7f805b7fe640 10 monclient: dump: 2024-03-29T10:48:17.798 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:17.798 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:17.798 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:17.798 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:17.798 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:17.798 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:17.798 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:17.798 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:17.798 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:17.798 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:17.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.812+0000 7f8061962640 10 monclient: shutdown 2024-03-29T10:48:17.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.812+0000 7f8061962640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:17.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.812+0000 7f8061962640 1 -- 172.21.15.98:0/1798548414 >> v1:172.21.15.98:6790/0 conn(0x7f805402b100 legacy=0x7f80540cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:17.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.813+0000 7f8061962640 1 -- 172.21.15.98:0/1798548414 shutdown_connections 2024-03-29T10:48:17.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.813+0000 7f8061962640 1 -- 172.21.15.98:0/1798548414 >> 172.21.15.98:0/1798548414 conn(0x7f80540058f0 msgr2=0x7f8054005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:17.800 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.814+0000 7f8061962640 1 -- 172.21.15.98:0/1798548414 shutdown_connections 2024-03-29T10:48:17.800 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.814+0000 7f8061962640 1 -- 172.21.15.98:0/1798548414 wait complete. 2024-03-29T10:48:17.800 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.815+0000 7f8061962640 10 monclient: build_initial_monmap 2024-03-29T10:48:17.800 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.815+0000 7f8061962640 10 monclient: monmap: 2024-03-29T10:48:17.800 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:17.800 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:17.800 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:48:17.800 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:48:17.800 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:17.800 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:17.800 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:17.800 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:48:17.801 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:48:17.801 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:17.801 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.815+0000 7f8061962640 1 librados: starting msgr at 2024-03-29T10:48:17.801 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.815+0000 7f8061962640 1 librados: starting objecter 2024-03-29T10:48:17.801 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.816+0000 7f8061962640 1 Processor -- start 2024-03-29T10:48:17.801 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.816+0000 7f8061962640 1 -- start start 2024-03-29T10:48:17.801 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.816+0000 7f8061962640 1 librados: setting wanted keys 2024-03-29T10:48:17.801 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.816+0000 7f8061962640 1 librados: calling monclient init 2024-03-29T10:48:17.801 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.816+0000 7f8061962640 10 monclient: init 2024-03-29T10:48:17.801 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.817+0000 7f8061962640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:17.801 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.817+0000 7f8061962640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:17.802 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.820+0000 7f8060960640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f8054006a70 0x7f80541644c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:49012/0 (socket says 172.21.15.98:49012) 2024-03-29T10:48:17.802 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.820+0000 7f8060960640 1 -- 172.21.15.98:0/2192509555 learned_addr learned my addr 172.21.15.98:0/2192509555 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:17.802 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.818+0000 7f8061962640 10 monclient(hunting): picked mon.noname-c con 0x7f8054006a70 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:17.802 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.820+0000 7f8061962640 10 monclient(hunting): picked mon.noname-a con 0x7f8054029c30 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:17.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.820+0000 7f8061962640 10 monclient(hunting): picked mon.noname-b con 0x7f80540c2d90 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:17.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.820+0000 7f8061962640 1 -- 172.21.15.98:0/2192509555 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8054164f90 con 0x7f8054029c30 2024-03-29T10:48:17.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.820+0000 7f8061962640 1 -- 172.21.15.98:0/2192509555 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805402f620 con 0x7f80540c2d90 2024-03-29T10:48:17.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.820+0000 7f8061962640 1 -- 172.21.15.98:0/2192509555 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8054164bd0 con 0x7f8054006a70 2024-03-29T10:48:17.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.820+0000 7f8061962640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:17.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.820+0000 7f8061962640 10 monclient(hunting): authenticate will time out at 1118.017734s 2024-03-29T10:48:17.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.821+0000 7f80597fa640 10 client.?.objecter ms_handle_connect 0x7f8054006a70 2024-03-29T10:48:17.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.821+0000 7f80597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:17.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.824+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101534537 0 0) 0x7f8054164bd0 con 0x7f8054006a70 2024-03-29T10:48:17.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.824+0000 7f80597fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:17.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.824+0000 7f80597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:17.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.824+0000 7f80597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:17.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.824+0000 7f80597fa640 10 monclient(hunting): my global_id is 4358 2024-03-29T10:48:17.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.824+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8040004eb0 con 0x7f8054006a70 2024-03-29T10:48:17.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.825+0000 7f80597fa640 10 client.?.objecter ms_handle_connect 0x7f8054029c30 2024-03-29T10:48:17.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.825+0000 7f80597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:17.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.825+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398666746 0 0) 0x7f8054164f90 con 0x7f8054029c30 2024-03-29T10:48:17.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.825+0000 7f80597fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:17.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.825+0000 7f80597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:17.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.825+0000 7f80597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:17.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.825+0000 7f80597fa640 10 monclient(hunting): my global_id is 4353 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.825+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80400051d0 con 0x7f8054029c30 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.825+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 968720522 0 0) 0x7f80400051d0 con 0x7f8054029c30 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.825+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8054164f90 con 0x7f8054029c30 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.826+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8050001630 con 0x7f8054029c30 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.826+0000 7f80597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.826+0000 7f80597fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.826+0000 7f80597fa640 10 monclient(hunting): dump: 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:17.809 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:17.810 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:17.810 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:17.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.826+0000 7f80597fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:17.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 client.?.objecter ms_handle_connect 0x7f80540c2d90 2024-03-29T10:48:17.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706711678 0 0) 0x7f805402f620 con 0x7f80540c2d90 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80400059c0 con 0x7f80540c2d90 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 139843590 0 0) 0x7f8054164f90 con 0x7f8054029c30 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 >> v1:172.21.15.98:6790/0 conn(0x7f8054006a70 legacy=0x7f80541644c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 >> v1:172.21.15.189:6789/0 conn(0x7f80540c2d90 legacy=0x7f805402d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 monclient: _finish_hunting 0 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 1 monclient: found mon.a 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:48:17.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8054031bb0 con 0x7f8054029c30 2024-03-29T10:48:17.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 monclient: _finish_auth 0 2024-03-29T10:48:17.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 monclient: _check_auth_tickets 2024-03-29T10:48:17.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:17.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8050004a90 con 0x7f8054029c30 2024-03-29T10:48:17.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:17.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8050001630 con 0x7f8054029c30 2024-03-29T10:48:17.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:17.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:17.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f80597fa640 10 monclient: dump: 2024-03-29T10:48:17.812 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:17.812 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f805a7fc640 4 set_mon_vals no callback set 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f8061962640 5 monclient: authenticate success, global_id 4353 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f8061962640 10 monclient: _renew_subs 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.827+0000 7f8061962640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.828+0000 7f8061962640 1 -- 172.21.15.98:0/2192509555 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8054031da0 con 0x7f8054029c30 2024-03-29T10:48:17.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.828+0000 7f8061962640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:17.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.828+0000 7f8061962640 10 monclient: _renew_subs 2024-03-29T10:48:17.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.828+0000 7f8061962640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:48:17.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.828+0000 7f8061962640 1 -- 172.21.15.98:0/2192509555 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f805402d940 con 0x7f8054029c30 2024-03-29T10:48:17.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.828+0000 7f8061962640 1 librados: init done 2024-03-29T10:48:17.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.829+0000 7f8046ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:17.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.829+0000 7f8046ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:17.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.829+0000 7f8046ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:48:17.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.829+0000 7f8046ffd640 1 -- 172.21.15.98:0/2192509555 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8054164d60 con 0x7f8054029c30 2024-03-29T10:48:17.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.830+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f8050004cc0 con 0x7f8054029c30 2024-03-29T10:48:17.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.830+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f8050084620 con 0x7f8054029c30 2024-03-29T10:48:17.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.830+0000 7f80597fa640 10 client.4353.objecter ms_dispatch 0x7f805c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:17.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.830+0000 7f80597fa640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:17.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.830+0000 7f80597fa640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:17.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.831+0000 7f80597fa640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T10:48:17.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.831+0000 7f8060960640 10 monclient: get_auth_request con 0x7f804006e380 auth_method 0 2024-03-29T10:48:17.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.831+0000 7f80597fa640 10 client.4353.objecter ms_handle_connect 0x7f804006e380 2024-03-29T10:48:17.821 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.839+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8054164d60 con 0x7f8054029c30 2024-03-29T10:48:17.821 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.839+0000 7f80597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:17.821 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:17.839+0000 7f80597fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:18.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.170+0000 7f8046ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:18.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.170+0000 7f8046ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:18.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.170+0000 7f8046ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:48:18.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.170+0000 7f8046ffd640 1 -- 172.21.15.98:0/2192509555 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8054164d60 con 0x7f8054029c30 2024-03-29T10:48:18.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.171+0000 7f80597fa640 1 -- 172.21.15.98:0/2192509555 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f8054164d60 con 0x7f8054029c30 2024-03-29T10:48:18.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.171+0000 7f80597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:18.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.171+0000 7f80597fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:18.154 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:48:18.154 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:18.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.175+0000 7f8046ffd640 10 librados: watch_flush enter 2024-03-29T10:48:18.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.175+0000 7f8046ffd640 10 librados: watch_flush exit 2024-03-29T10:48:18.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.175+0000 7f8046ffd640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T10:48:18.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.175+0000 7f8046ffd640 10 client.4353.objecter successfully canceled tick 2024-03-29T10:48:18.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.175+0000 7f8046ffd640 1 -- 172.21.15.98:0/2192509555 >> v1:172.21.15.98:6816/1481799426 conn(0x7f804006e380 legacy=0x7f8040070840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:18.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.175+0000 7f8046ffd640 10 monclient: shutdown 2024-03-29T10:48:18.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.176+0000 7f8046ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:18.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.176+0000 7f8046ffd640 1 -- 172.21.15.98:0/2192509555 >> v1:172.21.15.98:6789/0 conn(0x7f8054029c30 legacy=0x7f80540c2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:18.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.176+0000 7f8046ffd640 1 -- 172.21.15.98:0/2192509555 shutdown_connections 2024-03-29T10:48:18.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.176+0000 7f8046ffd640 1 -- 172.21.15.98:0/2192509555 >> 172.21.15.98:0/2192509555 conn(0x7f80540058f0 msgr2=0x7f80540c4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:18.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.177+0000 7f8046ffd640 1 -- 172.21.15.98:0/2192509555 shutdown_connections 2024-03-29T10:48:18.160 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.177+0000 7f8046ffd640 1 -- 172.21.15.98:0/2192509555 wait complete. 2024-03-29T10:48:18.160 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:18.177+0000 7f8046ffd640 1 librados: shutdown 2024-03-29T10:48:18.176 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:24.984 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:25.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.149+0000 7f8a20cf1640 10 monclient: get_monmap_and_config 2024-03-29T10:48:25.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.149+0000 7f8a20cf1640 10 monclient: build_initial_monmap 2024-03-29T10:48:25.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.149+0000 7f8a20cf1640 10 monclient: monmap: 2024-03-29T10:48:25.133 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:25.133 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:25.133 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:48:25.150711+0000 2024-03-29T10:48:25.133 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:48:25.150711+0000 2024-03-29T10:48:25.133 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:25.133 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:25.133 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:25.133 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:48:25.133 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:48:25.133 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:25.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.151+0000 7f8a20cf1640 1 Processor -- start 2024-03-29T10:48:25.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.151+0000 7f8a20cf1640 1 -- start start 2024-03-29T10:48:25.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.151+0000 7f8a20cf1640 10 monclient: init 2024-03-29T10:48:25.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.152+0000 7f8a20cf1640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:25.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.152+0000 7f8a20cf1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:25.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.152+0000 7f8a20cf1640 10 monclient(hunting): picked mon.noname-b con 0x7f8a1c0b1f60 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:25.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.152+0000 7f8a20cf1640 10 monclient(hunting): picked mon.noname-a con 0x7f8a1c071eb0 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:25.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.152+0000 7f8a20cf1640 10 monclient(hunting): picked mon.noname-c con 0x7f8a1c143d10 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:25.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.152+0000 7f8a20cf1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c06ef80 con 0x7f8a1c071eb0 2024-03-29T10:48:25.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.152+0000 7f8a20cf1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c06f260 con 0x7f8a1c143d10 2024-03-29T10:48:25.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.152+0000 7f8a20cf1640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c0b40c0 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.152+0000 7f8a20cf1640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:25.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.152+0000 7f8a20cf1640 10 monclient(hunting): authenticate will time out at 1125.349511s 2024-03-29T10:48:25.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.153+0000 7f8a1a59c640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f8a1c0b1f60 0x7f8a1c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:55190/0 (socket says 172.21.15.98:55190) 2024-03-29T10:48:25.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.153+0000 7f8a1a59c640 1 -- 172.21.15.98:0/1020740673 learned_addr learned my addr 172.21.15.98:0/1020740673 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:25.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.153+0000 7f8a1959a640 1 -- 172.21.15.98:0/1020740673 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067729362 0 0) 0x7f8a1c0b40c0 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.153+0000 7f8a1959a640 10 monclient(hunting): _negotiate 2024-03-29T10:48:25.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.153+0000 7f8a1959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:25.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.153+0000 7f8a1959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:25.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.153+0000 7f8a1959a640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:48:25.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.153+0000 7f8a1959a640 1 -- 172.21.15.98:0/1020740673 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a04003520 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.154+0000 7f8a1959a640 1 -- 172.21.15.98:0/1020740673 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2316733464 0 0) 0x7f8a04003520 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.154+0000 7f8a1959a640 1 -- 172.21.15.98:0/1020740673 >> v1:172.21.15.189:6789/0 conn(0x7f8a1c143d10 legacy=0x7f8a1c146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:25.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.154+0000 7f8a1959a640 1 -- 172.21.15.98:0/1020740673 >> v1:172.21.15.98:6789/0 conn(0x7f8a1c071eb0 legacy=0x7f8a1c146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:25.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a1959a640 10 monclient: _finish_hunting 0 2024-03-29T10:48:25.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a1959a640 1 monclient: found mon.noname-b 2024-03-29T10:48:25.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a1959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:25.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a1959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:48:25.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a1959a640 1 -- 172.21.15.98:0/1020740673 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c155d70 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a1959a640 10 monclient: _finish_auth 0 2024-03-29T10:48:25.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a1959a640 10 monclient: _check_auth_tickets 2024-03-29T10:48:25.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a1959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:25.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a1959a640 1 -- 172.21.15.98:0/1020740673 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8a10001670 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a1959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a1959a640 10 monclient: dump: 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a20cf1640 5 monclient: authenticate success, global_id 4361 2024-03-29T10:48:25.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.155+0000 7f8a20cf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.156+0000 7f8a1959a640 1 -- 172.21.15.98:0/1020740673 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a100018d0 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.156+0000 7f8a1959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.156+0000 7f8a20cf1640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.156+0000 7f8a20cf1640 4 set_mon_vals no callback set 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.156+0000 7f8a1959a640 1 -- 172.21.15.98:0/1020740673 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8a10001d20 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.156+0000 7f8a1959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.156+0000 7f8a1959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.156+0000 7f8a1959a640 10 monclient: dump: 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:25.145 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.157+0000 7f8a20cf1640 10 monclient: shutdown 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.157+0000 7f8a20cf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.157+0000 7f8a20cf1640 1 -- 172.21.15.98:0/1020740673 >> v1:172.21.15.98:6790/0 conn(0x7f8a1c0b1f60 legacy=0x7f8a1c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.157+0000 7f8a20cf1640 1 -- 172.21.15.98:0/1020740673 shutdown_connections 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.157+0000 7f8a20cf1640 1 -- 172.21.15.98:0/1020740673 >> 172.21.15.98:0/1020740673 conn(0x7f8a1c066a00 msgr2=0x7f8a1c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.157+0000 7f8a20cf1640 1 -- 172.21.15.98:0/1020740673 shutdown_connections 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.157+0000 7f8a20cf1640 1 -- 172.21.15.98:0/1020740673 wait complete. 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.158+0000 7f8a20cf1640 10 monclient: build_initial_monmap 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.159+0000 7f8a20cf1640 10 monclient: monmap: 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:25.146 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.159+0000 7f8a20cf1640 1 librados: starting msgr at 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.159+0000 7f8a20cf1640 1 librados: starting objecter 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.159+0000 7f8a20cf1640 1 Processor -- start 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.160+0000 7f8a20cf1640 1 -- start start 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.160+0000 7f8a20cf1640 1 librados: setting wanted keys 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.160+0000 7f8a20cf1640 1 librados: calling monclient init 2024-03-29T10:48:25.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.160+0000 7f8a20cf1640 10 monclient: init 2024-03-29T10:48:25.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.161+0000 7f8a20cf1640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:25.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.161+0000 7f8a20cf1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:25.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.162+0000 7f8a20cf1640 10 monclient(hunting): picked mon.noname-a con 0x7f8a1c0b1f60 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:25.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.162+0000 7f8a20cf1640 10 monclient(hunting): picked mon.noname-c con 0x7f8a1c143d10 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:25.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.162+0000 7f8a20cf1640 10 monclient(hunting): picked mon.noname-b con 0x7f8a1c1493d0 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:25.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.162+0000 7f8a20cf1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c14ad60 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.162+0000 7f8a20cf1640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c1498b0 con 0x7f8a1c1493d0 2024-03-29T10:48:25.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.162+0000 7f8a20cf1640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c14afb0 con 0x7f8a1c143d10 2024-03-29T10:48:25.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.162+0000 7f8a20cf1640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:25.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.162+0000 7f8a20cf1640 10 monclient(hunting): authenticate will time out at 1125.359925s 2024-03-29T10:48:25.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.163+0000 7f8a19d9b640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f8a1c143d10 0x7f8a1c13b030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:57894/0 (socket says 172.21.15.98:57894) 2024-03-29T10:48:25.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.163+0000 7f8a19d9b640 1 -- 172.21.15.98:0/2731266268 learned_addr learned my addr 172.21.15.98:0/2731266268 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:25.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.166+0000 7f8a0affd640 10 client.?.objecter ms_handle_connect 0x7f8a1c143d10 2024-03-29T10:48:25.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.166+0000 7f8a0affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:25.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.166+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019878693 0 0) 0x7f8a1c14afb0 con 0x7f8a1c143d10 2024-03-29T10:48:25.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.166+0000 7f8a0affd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.166+0000 7f8a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.166+0000 7f8a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.166+0000 7f8a0affd640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.166+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89fc004eb0 con 0x7f8a1c143d10 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.166+0000 7f8a0affd640 10 client.?.objecter ms_handle_connect 0x7f8a1c0b1f60 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.166+0000 7f8a0affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031077288 0 0) 0x7f8a1c14ad60 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a1c14afb0 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 client.?.objecter ms_handle_connect 0x7f8a1c1493d0 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:25.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3693918477 0 0) 0x7f89fc004eb0 con 0x7f8a1c143d10 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a1c14ad60 con 0x7f8a1c143d10 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8a0c001960 con 0x7f8a1c143d10 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): dump: 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:25.153 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410257056 0 0) 0x7f8a1c1498b0 con 0x7f8a1c1493d0 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): my global_id is 4360 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89fc004eb0 con 0x7f8a1c1493d0 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3900819521 0 0) 0x7f8a1c14afb0 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a1c1498b0 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8a10001630 con 0x7f8a1c0b1f60 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:25.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.167+0000 7f8a0affd640 10 monclient(hunting): dump: 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 756422214 0 0) 0x7f8a1c14ad60 con 0x7f8a1c143d10 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 >> v1:172.21.15.189:6789/0 conn(0x7f8a1c1493d0 legacy=0x7f8a1c1e6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 >> v1:172.21.15.98:6789/0 conn(0x7f8a1c0b1f60 legacy=0x7f8a1c13f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:25.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 1 monclient: found mon.c 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c14b290 con 0x7f8a1c143d10 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 10 monclient: _finish_auth 0 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a20cf1640 5 monclient: authenticate success, global_id 4364 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a20cf1640 10 monclient: _renew_subs 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a20cf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a20cf1640 1 -- 172.21.15.98:0/2731266268 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a1c149b00 con 0x7f8a1c143d10 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a20cf1640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a20cf1640 10 monclient: _renew_subs 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a20cf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a20cf1640 1 -- 172.21.15.98:0/2731266268 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1c154be0 con 0x7f8a1c143d10 2024-03-29T10:48:25.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a0c001ac0 con 0x7f8a1c143d10 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a20cf1640 1 librados: init done 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8a0c0038a0 con 0x7f8a1c143d10 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0affd640 10 monclient: dump: 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:25.157 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:25.158 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:25.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.168+0000 7f8a0bfff640 4 set_mon_vals no callback set 2024-03-29T10:48:25.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.170+0000 7f8a20cf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:25.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.170+0000 7f8a20cf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:25.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.170+0000 7f8a20cf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:25.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.170+0000 7f8a20cf1640 1 -- 172.21.15.98:0/2731266268 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1c075f10 con 0x7f8a1c143d10 2024-03-29T10:48:25.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.170+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f8a0c003a00 con 0x7f8a1c143d10 2024-03-29T10:48:25.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.171+0000 7f8a1a59c640 10 monclient: get_auth_request con 0x7f89fc06e000 auth_method 0 2024-03-29T10:48:25.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.172+0000 7f8a0affd640 10 client.4364.objecter ms_handle_connect 0x7f89fc06e000 2024-03-29T10:48:25.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.172+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f8a0c084a70 con 0x7f8a1c143d10 2024-03-29T10:48:25.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.172+0000 7f8a0affd640 10 client.4364.objecter ms_dispatch 0x7f8a1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:25.160 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.172+0000 7f8a0affd640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:25.160 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.172+0000 7f8a0affd640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:25.160 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.172+0000 7f8a0affd640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-29T10:48:25.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.179+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a0c050180 con 0x7f8a1c143d10 2024-03-29T10:48:25.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.179+0000 7f8a0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:25.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.179+0000 7f8a0affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:25.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.510+0000 7f8a20cf1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:25.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.510+0000 7f8a20cf1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:25.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.510+0000 7f8a20cf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:25.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.510+0000 7f8a20cf1640 1 -- 172.21.15.98:0/2731266268 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a1c149c90 con 0x7f8a1c143d10 2024-03-29T10:48:25.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.513+0000 7f8a0affd640 1 -- 172.21.15.98:0/2731266268 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f8a0c052280 con 0x7f8a1c143d10 2024-03-29T10:48:25.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.513+0000 7f8a0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:25.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.513+0000 7f8a0affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:25.495 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:48:25.496 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:25.499 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.517+0000 7f8a20cf1640 10 librados: watch_flush enter 2024-03-29T10:48:25.500 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.518+0000 7f8a20cf1640 10 librados: watch_flush exit 2024-03-29T10:48:25.500 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.518+0000 7f8a20cf1640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-29T10:48:25.500 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.518+0000 7f8a20cf1640 10 client.4364.objecter successfully canceled tick 2024-03-29T10:48:25.500 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.518+0000 7f8a20cf1640 1 -- 172.21.15.98:0/2731266268 >> v1:172.21.15.98:6816/1481799426 conn(0x7f89fc06e000 legacy=0x7f89fc0704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:25.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.518+0000 7f8a20cf1640 10 monclient: shutdown 2024-03-29T10:48:25.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.518+0000 7f8a20cf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:25.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.518+0000 7f8a20cf1640 1 -- 172.21.15.98:0/2731266268 >> v1:172.21.15.98:6790/0 conn(0x7f8a1c143d10 legacy=0x7f8a1c13b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:25.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.519+0000 7f8a20cf1640 1 -- 172.21.15.98:0/2731266268 shutdown_connections 2024-03-29T10:48:25.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.519+0000 7f8a20cf1640 1 -- 172.21.15.98:0/2731266268 >> 172.21.15.98:0/2731266268 conn(0x7f8a1c066a00 msgr2=0x7f8a1c142870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:25.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.519+0000 7f8a20cf1640 1 -- 172.21.15.98:0/2731266268 shutdown_connections 2024-03-29T10:48:25.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.519+0000 7f8a20cf1640 1 -- 172.21.15.98:0/2731266268 wait complete. 2024-03-29T10:48:25.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:25.520+0000 7f8a20cf1640 1 librados: shutdown 2024-03-29T10:48:25.520 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:32.328 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:32.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.496+0000 7fabc7a4a640 10 monclient: get_monmap_and_config 2024-03-29T10:48:32.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.496+0000 7fabc7a4a640 10 monclient: build_initial_monmap 2024-03-29T10:48:32.479 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.496+0000 7fabc7a4a640 10 monclient: monmap: 2024-03-29T10:48:32.479 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:32.479 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:32.479 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:48:32.496974+0000 2024-03-29T10:48:32.479 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:48:32.496974+0000 2024-03-29T10:48:32.479 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:32.479 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:32.479 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:32.479 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:48:32.479 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:48:32.479 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:32.480 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.498+0000 7fabc7a4a640 1 Processor -- start 2024-03-29T10:48:32.480 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.499+0000 7fabc7a4a640 1 -- start start 2024-03-29T10:48:32.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.499+0000 7fabc7a4a640 10 monclient: init 2024-03-29T10:48:32.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.499+0000 7fabc7a4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:32.481 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.499+0000 7fabc7a4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:32.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.499+0000 7fabc7a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fabc01422b0 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:32.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.499+0000 7fabc7a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fabc0142690 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:32.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.499+0000 7fabc7a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fabc00b17d0 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:32.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.499+0000 7fabc7a4a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc0142c80 con 0x7fabc00b17d0 2024-03-29T10:48:32.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.499+0000 7fabc7a4a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc00aae40 con 0x7fabc0142690 2024-03-29T10:48:32.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.499+0000 7fabc7a4a640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc013fbd0 con 0x7fabc01422b0 2024-03-29T10:48:32.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.499+0000 7fabc7a4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:32.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.499+0000 7fabc7a4a640 10 monclient(hunting): authenticate will time out at 1132.696897s 2024-03-29T10:48:32.485 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabc57e6640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fabc01422b0 0x7fabc013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:45908/0 (socket says 172.21.15.98:45908) 2024-03-29T10:48:32.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabc57e6640 1 -- 172.21.15.98:0/3371313924 learned_addr learned my addr 172.21.15.98:0/3371313924 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:32.488 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 1 -- 172.21.15.98:0/3371313924 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532699332 0 0) 0x7fabc013fbd0 con 0x7fabc01422b0 2024-03-29T10:48:32.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:32.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:32.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:32.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:48:32.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 1 -- 172.21.15.98:0/3371313924 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabac003520 con 0x7fabc01422b0 2024-03-29T10:48:32.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 1 -- 172.21.15.98:0/3371313924 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643338647 0 0) 0x7fabc0142c80 con 0x7fabc00b17d0 2024-03-29T10:48:32.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:32.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:32.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:32.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 10 monclient(hunting): my global_id is 4359 2024-03-29T10:48:32.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 1 -- 172.21.15.98:0/3371313924 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabac003840 con 0x7fabc00b17d0 2024-03-29T10:48:32.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 1 -- 172.21.15.98:0/3371313924 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 39140799 0 0) 0x7fabac003520 con 0x7fabc01422b0 2024-03-29T10:48:32.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 1 -- 172.21.15.98:0/3371313924 >> v1:172.21.15.189:6789/0 conn(0x7fabc0142690 legacy=0x7fabc00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 1 -- 172.21.15.98:0/3371313924 >> v1:172.21.15.98:6789/0 conn(0x7fabc00b17d0 legacy=0x7fabc00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.504+0000 7fabb7fff640 1 monclient: found mon.noname-b 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 1 -- 172.21.15.98:0/3371313924 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc0151940 con 0x7fabc01422b0 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 10 monclient: _finish_auth 0 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 1 -- 172.21.15.98:0/3371313924 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fabb80019f0 con 0x7fabc01422b0 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabc7a4a640 5 monclient: authenticate success, global_id 4367 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:32.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 10 monclient: dump: 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabc7a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 1 -- 172.21.15.98:0/3371313924 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabb8001bd0 con 0x7fabc01422b0 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 1 -- 172.21.15.98:0/3371313924 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fabb8001f60 con 0x7fabc01422b0 2024-03-29T10:48:32.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabb7fff640 10 monclient: dump: 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabc7a4a640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.505+0000 7fabc7a4a640 4 set_mon_vals no callback set 2024-03-29T10:48:32.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.506+0000 7fabc7a4a640 10 monclient: shutdown 2024-03-29T10:48:32.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.506+0000 7fabc7a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:32.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.506+0000 7fabc7a4a640 1 -- 172.21.15.98:0/3371313924 >> v1:172.21.15.98:6790/0 conn(0x7fabc01422b0 legacy=0x7fabc013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:32.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.506+0000 7fabc7a4a640 1 -- 172.21.15.98:0/3371313924 shutdown_connections 2024-03-29T10:48:32.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.506+0000 7fabc7a4a640 1 -- 172.21.15.98:0/3371313924 >> 172.21.15.98:0/3371313924 conn(0x7fabc0066a00 msgr2=0x7fabc0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:32.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.507+0000 7fabc7a4a640 1 -- 172.21.15.98:0/3371313924 shutdown_connections 2024-03-29T10:48:32.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.507+0000 7fabc7a4a640 1 -- 172.21.15.98:0/3371313924 wait complete. 2024-03-29T10:48:32.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.507+0000 7fabc7a4a640 10 monclient: build_initial_monmap 2024-03-29T10:48:32.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.507+0000 7fabc7a4a640 10 monclient: monmap: 2024-03-29T10:48:32.494 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.508+0000 7fabc7a4a640 1 librados: starting msgr at 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.508+0000 7fabc7a4a640 1 librados: starting objecter 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.508+0000 7fabc7a4a640 1 Processor -- start 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.508+0000 7fabc7a4a640 1 -- start start 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.508+0000 7fabc7a4a640 1 librados: setting wanted keys 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.508+0000 7fabc7a4a640 1 librados: calling monclient init 2024-03-29T10:48:32.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.509+0000 7fabc7a4a640 10 monclient: init 2024-03-29T10:48:32.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.509+0000 7fabc7a4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:32.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.510+0000 7fabc7a4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:32.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.510+0000 7fabc7a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fabc00b17d0 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:32.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.513+0000 7fabc7a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fabc01422b0 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:32.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.513+0000 7fabc7a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fabc0142690 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:32.497 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.513+0000 7fabc7a4a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc00afee0 con 0x7fabc0142690 2024-03-29T10:48:32.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.513+0000 7fabc7a4a640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc01356e0 con 0x7fabc00b17d0 2024-03-29T10:48:32.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.513+0000 7fabc4fe5640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fabc01422b0 0x7fabc01348c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:47134/0 (socket says 172.21.15.98:47134) 2024-03-29T10:48:32.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.513+0000 7fabc4fe5640 1 -- 172.21.15.98:0/399748180 learned_addr learned my addr 172.21.15.98:0/399748180 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:32.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.513+0000 7fabc7a4a640 1 -- 172.21.15.98:0/399748180 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc00b0d70 con 0x7fabc01422b0 2024-03-29T10:48:32.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.513+0000 7fabb5ffb640 10 client.?.objecter ms_handle_connect 0x7fabc01422b0 2024-03-29T10:48:32.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.513+0000 7fabb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:32.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.513+0000 7fabb5ffb640 10 client.?.objecter ms_handle_connect 0x7fabc00b17d0 2024-03-29T10:48:32.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.513+0000 7fabb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:32.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.515+0000 7fabc7a4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:32.498 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.515+0000 7fabc7a4a640 10 monclient(hunting): authenticate will time out at 1132.712131s 2024-03-29T10:48:32.500 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.517+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102250093 0 0) 0x7fabc01356e0 con 0x7fabc00b17d0 2024-03-29T10:48:32.500 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.517+0000 7fabb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:32.500 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.517+0000 7fabb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.517+0000 7fabb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.517+0000 7fabb5ffb640 10 monclient(hunting): my global_id is 4363 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.517+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faba4004eb0 con 0x7fabc00b17d0 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.517+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953985984 0 0) 0x7fabc00b0d70 con 0x7fabc01422b0 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.517+0000 7fabb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.517+0000 7fabb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.517+0000 7fabb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.517+0000 7fabb5ffb640 10 monclient(hunting): my global_id is 4370 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.517+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc01356e0 con 0x7fabc01422b0 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.518+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2511076 0 0) 0x7faba4004eb0 con 0x7fabc00b17d0 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.518+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faba40054e0 con 0x7fabc00b17d0 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.518+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fabb8001f60 con 0x7fabc00b17d0 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.518+0000 7fabb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:32.501 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.518+0000 7fabb5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.518+0000 7fabb5ffb640 10 monclient(hunting): dump: 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.518+0000 7fabb5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.518+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1675565758 0 0) 0x7fabc01356e0 con 0x7fabc01422b0 2024-03-29T10:48:32.502 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.518+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faba4004eb0 con 0x7fabc01422b0 2024-03-29T10:48:32.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.518+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 224758025 0 0) 0x7faba40054e0 con 0x7fabc00b17d0 2024-03-29T10:48:32.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.518+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 >> v1:172.21.15.98:6790/0 conn(0x7fabc01422b0 legacy=0x7fabc01348c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:32.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.518+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 >> v1:172.21.15.98:6789/0 conn(0x7fabc0142690 legacy=0x7fabc0134fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:32.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:32.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 1 monclient: found mon.b 2024-03-29T10:48:32.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:32.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:48:32.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc00b1050 con 0x7fabc00b17d0 2024-03-29T10:48:32.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:32.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:32.503 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:32.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabb8004e10 con 0x7fabc00b17d0 2024-03-29T10:48:32.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:32.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fabb8005120 con 0x7fabc00b17d0 2024-03-29T10:48:32.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:32.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:32.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb5ffb640 10 monclient: dump: 2024-03-29T10:48:32.504 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:32.504 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:32.504 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:32.504 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabc7a4a640 5 monclient: authenticate success, global_id 4363 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabc7a4a640 10 monclient: _renew_subs 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabc7a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabc7a4a640 1 -- 172.21.15.98:0/399748180 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc00b1330 con 0x7fabc00b17d0 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabc7a4a640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabc7a4a640 10 monclient: _renew_subs 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.520+0000 7fabc7a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.520+0000 7fabc7a4a640 1 -- 172.21.15.98:0/399748180 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc00acd60 con 0x7fabc00b17d0 2024-03-29T10:48:32.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.519+0000 7fabb77fe640 4 set_mon_vals no callback set 2024-03-29T10:48:32.506 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.520+0000 7fabc7a4a640 1 librados: init done 2024-03-29T10:48:32.506 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.522+0000 7fabab7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:32.506 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.522+0000 7fabab7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:32.506 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.522+0000 7fabab7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:48:32.506 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.522+0000 7fabab7fe640 1 -- 172.21.15.98:0/399748180 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc0075f10 con 0x7fabc00b17d0 2024-03-29T10:48:32.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.523+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fabb801b0f0 con 0x7fabc00b17d0 2024-03-29T10:48:32.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.523+0000 7fabc4fe5640 10 monclient: get_auth_request con 0x7faba406e090 auth_method 0 2024-03-29T10:48:32.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.524+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fabb8084c50 con 0x7fabc00b17d0 2024-03-29T10:48:32.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.524+0000 7fabb5ffb640 10 client.4363.objecter ms_dispatch 0x7fabc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:32.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.524+0000 7fabb5ffb640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:32.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.524+0000 7fabb5ffb640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:32.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.524+0000 7fabb5ffb640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-29T10:48:32.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.524+0000 7fabb5ffb640 10 client.4363.objecter ms_handle_connect 0x7faba406e090 2024-03-29T10:48:32.513 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.531+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabb8050590 con 0x7fabc00b17d0 2024-03-29T10:48:32.513 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.531+0000 7fabb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:32.513 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.531+0000 7fabb5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:32.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.860+0000 7fabab7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:32.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.860+0000 7fabab7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:32.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.860+0000 7fabab7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:48:32.843 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.860+0000 7fabab7fe640 1 -- 172.21.15.98:0/399748180 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabc00b14c0 con 0x7fabc00b17d0 2024-03-29T10:48:32.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.863+0000 7fabb5ffb640 1 -- 172.21.15.98:0/399748180 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fabb8050590 con 0x7fabc00b17d0 2024-03-29T10:48:32.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.863+0000 7fabb5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:32.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.863+0000 7fabb5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:32.846 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:48:32.846 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:32.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.868+0000 7fabab7fe640 10 librados: watch_flush enter 2024-03-29T10:48:32.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.868+0000 7fabab7fe640 10 librados: watch_flush exit 2024-03-29T10:48:32.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.868+0000 7fabab7fe640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-29T10:48:32.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.868+0000 7fabab7fe640 10 client.4363.objecter successfully canceled tick 2024-03-29T10:48:32.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.868+0000 7fabab7fe640 1 -- 172.21.15.98:0/399748180 >> v1:172.21.15.98:6816/1481799426 conn(0x7faba406e090 legacy=0x7faba4070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:32.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.868+0000 7fabab7fe640 10 monclient: shutdown 2024-03-29T10:48:32.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.869+0000 7fabab7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:32.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.869+0000 7fabab7fe640 1 -- 172.21.15.98:0/399748180 >> v1:172.21.15.189:6789/0 conn(0x7fabc00b17d0 legacy=0x7fabc00ac650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:32.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.869+0000 7fabab7fe640 1 -- 172.21.15.98:0/399748180 shutdown_connections 2024-03-29T10:48:32.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.869+0000 7fabab7fe640 1 -- 172.21.15.98:0/399748180 >> 172.21.15.98:0/399748180 conn(0x7fabc0066a00 msgr2=0x7fabc013d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:32.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.869+0000 7fabab7fe640 1 -- 172.21.15.98:0/399748180 shutdown_connections 2024-03-29T10:48:32.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.869+0000 7fabab7fe640 1 -- 172.21.15.98:0/399748180 wait complete. 2024-03-29T10:48:32.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:32.869+0000 7fabab7fe640 1 librados: shutdown 2024-03-29T10:48:32.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:39.675 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:39.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.843+0000 7f7f521fb640 10 monclient: get_monmap_and_config 2024-03-29T10:48:39.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.843+0000 7f7f521fb640 10 monclient: build_initial_monmap 2024-03-29T10:48:39.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.843+0000 7f7f521fb640 10 monclient: monmap: 2024-03-29T10:48:39.826 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:39.827 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:39.827 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:48:39.843878+0000 2024-03-29T10:48:39.827 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:48:39.843878+0000 2024-03-29T10:48:39.827 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:39.827 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:39.827 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:39.827 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:48:39.827 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:48:39.827 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:39.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.845+0000 7f7f521fb640 1 Processor -- start 2024-03-29T10:48:39.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.845+0000 7f7f521fb640 1 -- start start 2024-03-29T10:48:39.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.845+0000 7f7f521fb640 10 monclient: init 2024-03-29T10:48:39.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.846+0000 7f7f521fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:39.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.846+0000 7f7f521fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:39.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.846+0000 7f7f521fb640 10 monclient(hunting): picked mon.noname-a con 0x7f7f4c0b4330 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:39.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.846+0000 7f7f521fb640 10 monclient(hunting): picked mon.noname-b con 0x7f7f4c1470b0 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:39.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.846+0000 7f7f521fb640 10 monclient(hunting): picked mon.noname-c con 0x7f7f4c1439b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:39.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.846+0000 7f7f521fb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c06f2a0 con 0x7f7f4c0b4330 2024-03-29T10:48:39.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.846+0000 7f7f521fb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c146b90 con 0x7f7f4c1439b0 2024-03-29T10:48:39.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.846+0000 7f7f521fb640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c0b2ca0 con 0x7f7f4c1470b0 2024-03-29T10:48:39.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.846+0000 7f7f521fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:39.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.846+0000 7f7f521fb640 10 monclient(hunting): authenticate will time out at 1140.043832s 2024-03-29T10:48:39.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.849+0000 7f7f4affd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f7f4c1470b0 0x7f7f4c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:59813/0 (socket says 172.21.15.98:59813) 2024-03-29T10:48:39.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.850+0000 7f7f4affd640 1 -- 172.21.15.98:0/1726150134 learned_addr learned my addr 172.21.15.98:0/1726150134 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:39.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 1 -- 172.21.15.98:0/1726150134 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636863214 0 0) 0x7f7f4c0b2ca0 con 0x7f7f4c1470b0 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 1 -- 172.21.15.98:0/1726150134 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f34003520 con 0x7f7f4c1470b0 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 1 -- 172.21.15.98:0/1726150134 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 642542530 0 0) 0x7f7f34003520 con 0x7f7f4c1470b0 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 1 -- 172.21.15.98:0/1726150134 >> v1:172.21.15.189:6789/0 conn(0x7f7f4c1439b0 legacy=0x7f7f4c151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 1 -- 172.21.15.98:0/1726150134 >> v1:172.21.15.98:6789/0 conn(0x7f7f4c0b4330 legacy=0x7f7f4c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 1 monclient: found mon.noname-b 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.853+0000 7f7f4a7fc640 1 -- 172.21.15.98:0/1726150134 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c06e0c0 con 0x7f7f4c1470b0 2024-03-29T10:48:39.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f4a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f4a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f4a7fc640 1 -- 172.21.15.98:0/1726150134 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f7f40001690 con 0x7f7f4c1470b0 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f4a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f4a7fc640 10 monclient: dump: 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:39.840 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f521fb640 5 monclient: authenticate success, global_id 4373 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f521fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f4a7fc640 1 -- 172.21.15.98:0/1726150134 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f400017f0 con 0x7f7f4c1470b0 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f4a7fc640 1 -- 172.21.15.98:0/1726150134 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f7f40001c40 con 0x7f7f4c1470b0 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f4a7fc640 10 monclient: dump: 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:39.841 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:39.842 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:39.842 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:39.842 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:39.842 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:39.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f521fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:39.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.854+0000 7f7f521fb640 4 set_mon_vals no callback set 2024-03-29T10:48:39.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.855+0000 7f7f521fb640 10 monclient: shutdown 2024-03-29T10:48:39.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.855+0000 7f7f521fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:39.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.855+0000 7f7f521fb640 1 -- 172.21.15.98:0/1726150134 >> v1:172.21.15.98:6790/0 conn(0x7f7f4c1470b0 legacy=0x7f7f4c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.858+0000 7f7f521fb640 1 -- 172.21.15.98:0/1726150134 shutdown_connections 2024-03-29T10:48:39.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.858+0000 7f7f521fb640 1 -- 172.21.15.98:0/1726150134 >> 172.21.15.98:0/1726150134 conn(0x7f7f4c066a00 msgr2=0x7f7f4c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:39.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.858+0000 7f7f521fb640 1 -- 172.21.15.98:0/1726150134 shutdown_connections 2024-03-29T10:48:39.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.858+0000 7f7f521fb640 1 -- 172.21.15.98:0/1726150134 wait complete. 2024-03-29T10:48:39.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.859+0000 7f7f521fb640 10 monclient: build_initial_monmap 2024-03-29T10:48:39.844 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.859+0000 7f7f521fb640 10 monclient: monmap: 2024-03-29T10:48:39.844 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:39.844 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:39.844 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:48:39.844 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:48:39.845 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:39.845 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:39.845 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:39.845 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:48:39.845 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:48:39.845 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:39.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.859+0000 7f7f521fb640 1 librados: starting msgr at 2024-03-29T10:48:39.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.859+0000 7f7f521fb640 1 librados: starting objecter 2024-03-29T10:48:39.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.860+0000 7f7f521fb640 1 Processor -- start 2024-03-29T10:48:39.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.860+0000 7f7f521fb640 1 -- start start 2024-03-29T10:48:39.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.860+0000 7f7f521fb640 1 librados: setting wanted keys 2024-03-29T10:48:39.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.860+0000 7f7f521fb640 1 librados: calling monclient init 2024-03-29T10:48:39.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.860+0000 7f7f521fb640 10 monclient: init 2024-03-29T10:48:39.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.861+0000 7f7f521fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:39.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.861+0000 7f7f521fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:39.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.863+0000 7f7f521fb640 10 monclient(hunting): picked mon.noname-b con 0x7f7f4c0b4330 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:39.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.867+0000 7f7f4affd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7f4c1439b0 0x7f7f4c11f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33153/0 (socket says 172.21.15.98:33153) 2024-03-29T10:48:39.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.867+0000 7f7f521fb640 10 monclient(hunting): picked mon.noname-a con 0x7f7f4c1439b0 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:39.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.867+0000 7f7f4affd640 1 -- 172.21.15.98:0/4013287291 learned_addr learned my addr 172.21.15.98:0/4013287291 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:39.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.867+0000 7f7f521fb640 10 monclient(hunting): picked mon.noname-c con 0x7f7f4c1470b0 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:39.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.867+0000 7f7f521fb640 1 -- 172.21.15.98:0/4013287291 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c1dbd60 con 0x7f7f4c1439b0 2024-03-29T10:48:39.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.867+0000 7f7f521fb640 1 -- 172.21.15.98:0/4013287291 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c11fe80 con 0x7f7f4c0b4330 2024-03-29T10:48:39.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.867+0000 7f7f521fb640 1 -- 172.21.15.98:0/4013287291 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4c120010 con 0x7f7f4c1470b0 2024-03-29T10:48:39.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.867+0000 7f7f521fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:39.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.867+0000 7f7f521fb640 10 monclient(hunting): authenticate will time out at 1140.064885s 2024-03-29T10:48:39.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.867+0000 7f7f33fff640 10 client.?.objecter ms_handle_connect 0x7f7f4c1439b0 2024-03-29T10:48:39.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.868+0000 7f7f33fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:39.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.868+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764048441 0 0) 0x7f7f4c1dbd60 con 0x7f7f4c1439b0 2024-03-29T10:48:39.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.868+0000 7f7f33fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:39.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.868+0000 7f7f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:39.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.868+0000 7f7f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.868+0000 7f7f33fff640 10 monclient(hunting): my global_id is 4365 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.868+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f38004eb0 con 0x7f7f4c1439b0 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.871+0000 7f7f33fff640 10 client.?.objecter ms_handle_connect 0x7f7f4c1470b0 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.871+0000 7f7f33fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.871+0000 7f7f33fff640 10 client.?.objecter ms_handle_connect 0x7f7f4c0b4330 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.871+0000 7f7f33fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742017140 0 0) 0x7f7f4c11fe80 con 0x7f7f4c0b4330 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 10 monclient(hunting): my global_id is 4369 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f4c1dbd60 con 0x7f7f4c0b4330 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2529800446 0 0) 0x7f7f38004eb0 con 0x7f7f4c1439b0 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f4c11fe80 con 0x7f7f4c1439b0 2024-03-29T10:48:39.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635371891 0 0) 0x7f7f4c120010 con 0x7f7f4c1470b0 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f38004eb0 con 0x7f7f4c1470b0 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.872+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f7f40001b40 con 0x7f7f4c1439b0 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.873+0000 7f7f33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.873+0000 7f7f33fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.873+0000 7f7f33fff640 10 monclient(hunting): dump: 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:39.859 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.873+0000 7f7f33fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.873+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3780773943 0 0) 0x7f7f4c1dbd60 con 0x7f7f4c0b4330 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.873+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f4c120010 con 0x7f7f4c0b4330 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.873+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 935406614 0 0) 0x7f7f4c11fe80 con 0x7f7f4c1439b0 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.873+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 >> v1:172.21.15.98:6790/0 conn(0x7f7f4c1470b0 legacy=0x7f7f4c1e5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.873+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 >> v1:172.21.15.189:6789/0 conn(0x7f7f4c0b4330 legacy=0x7f7f4c1db520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.873+0000 7f7f33fff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.873+0000 7f7f33fff640 1 monclient: found mon.a 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.873+0000 7f7f33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:39.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.873+0000 7f7f33fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:48:39.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.874+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c1dc1f0 con 0x7f7f4c1439b0 2024-03-29T10:48:39.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.874+0000 7f7f33fff640 10 monclient: _finish_auth 0 2024-03-29T10:48:39.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.874+0000 7f7f33fff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:39.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.874+0000 7f7f33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:39.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.874+0000 7f7f521fb640 5 monclient: authenticate success, global_id 4365 2024-03-29T10:48:39.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.874+0000 7f7f521fb640 10 monclient: _renew_subs 2024-03-29T10:48:39.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.874+0000 7f7f521fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:48:39.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.874+0000 7f7f521fb640 1 -- 172.21.15.98:0/4013287291 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4c1dc440 con 0x7f7f4c1439b0 2024-03-29T10:48:39.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.874+0000 7f7f521fb640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:39.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.874+0000 7f7f521fb640 10 monclient: _renew_subs 2024-03-29T10:48:39.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.874+0000 7f7f521fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:48:39.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.874+0000 7f7f521fb640 1 -- 172.21.15.98:0/4013287291 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4c1d06a0 con 0x7f7f4c1439b0 2024-03-29T10:48:39.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.874+0000 7f7f521fb640 1 librados: init done 2024-03-29T10:48:39.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.876+0000 7f7f521fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:39.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.876+0000 7f7f521fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:39.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.876+0000 7f7f521fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:48:39.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.876+0000 7f7f521fb640 1 -- 172.21.15.98:0/4013287291 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4c075f10 con 0x7f7f4c1439b0 2024-03-29T10:48:39.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.877+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f40004a90 con 0x7f7f4c1439b0 2024-03-29T10:48:39.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.877+0000 7f7f33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:39.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.877+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f7f40001ca0 con 0x7f7f4c1439b0 2024-03-29T10:48:39.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.877+0000 7f7f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:39.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.877+0000 7f7f33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:39.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.877+0000 7f7f33fff640 10 monclient: dump: 2024-03-29T10:48:39.863 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:39.864 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:39.864 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:39.864 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:39.864 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:39.864 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:39.864 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:39.864 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:39.864 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:39.864 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:39.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.877+0000 7f7f49ffb640 4 set_mon_vals no callback set 2024-03-29T10:48:39.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.882+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f7f4001ac10 con 0x7f7f4c1439b0 2024-03-29T10:48:39.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.883+0000 7f7f4b7fe640 10 monclient: get_auth_request con 0x7f7f3806de50 auth_method 0 2024-03-29T10:48:39.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.884+0000 7f7f33fff640 10 client.4365.objecter ms_handle_connect 0x7f7f3806de50 2024-03-29T10:48:39.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.884+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f7f40084a40 con 0x7f7f4c1439b0 2024-03-29T10:48:39.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.884+0000 7f7f33fff640 10 client.4365.objecter ms_dispatch 0x7f7f4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:39.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.884+0000 7f7f33fff640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:39.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.884+0000 7f7f33fff640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:39.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.884+0000 7f7f33fff640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-29T10:48:39.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.885+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f40050320 con 0x7f7f4c1439b0 2024-03-29T10:48:39.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.885+0000 7f7f33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:39.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:39.885+0000 7f7f33fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:40.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.214+0000 7f7f521fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:40.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.214+0000 7f7f521fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:40.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.214+0000 7f7f521fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:48:40.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.214+0000 7f7f521fb640 1 -- 172.21.15.98:0/4013287291 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f4c074f10 con 0x7f7f4c1439b0 2024-03-29T10:48:40.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.215+0000 7f7f33fff640 1 -- 172.21.15.98:0/4013287291 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f7f40052250 con 0x7f7f4c1439b0 2024-03-29T10:48:40.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.215+0000 7f7f33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:40.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.215+0000 7f7f33fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:40.198 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:48:40.198 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:40.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.218+0000 7f7f521fb640 10 librados: watch_flush enter 2024-03-29T10:48:40.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.219+0000 7f7f521fb640 10 librados: watch_flush exit 2024-03-29T10:48:40.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.219+0000 7f7f521fb640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-29T10:48:40.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.219+0000 7f7f521fb640 10 client.4365.objecter successfully canceled tick 2024-03-29T10:48:40.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.219+0000 7f7f521fb640 1 -- 172.21.15.98:0/4013287291 >> v1:172.21.15.98:6816/1481799426 conn(0x7f7f3806de50 legacy=0x7f7f38070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:40.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.219+0000 7f7f521fb640 10 monclient: shutdown 2024-03-29T10:48:40.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.219+0000 7f7f521fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:40.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.219+0000 7f7f521fb640 1 -- 172.21.15.98:0/4013287291 >> v1:172.21.15.98:6789/0 conn(0x7f7f4c1439b0 legacy=0x7f7f4c11f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:40.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.219+0000 7f7f521fb640 1 -- 172.21.15.98:0/4013287291 shutdown_connections 2024-03-29T10:48:40.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.219+0000 7f7f521fb640 1 -- 172.21.15.98:0/4013287291 >> 172.21.15.98:0/4013287291 conn(0x7f7f4c066a00 msgr2=0x7f7f4c1d15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:40.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.220+0000 7f7f521fb640 1 -- 172.21.15.98:0/4013287291 shutdown_connections 2024-03-29T10:48:40.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.220+0000 7f7f521fb640 1 -- 172.21.15.98:0/4013287291 wait complete. 2024-03-29T10:48:40.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:40.220+0000 7f7f521fb640 1 librados: shutdown 2024-03-29T10:48:40.219 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:47.026 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:47.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.204+0000 7fe7857d6640 10 monclient: get_monmap_and_config 2024-03-29T10:48:47.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.204+0000 7fe7857d6640 10 monclient: build_initial_monmap 2024-03-29T10:48:47.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.204+0000 7fe7857d6640 10 monclient: monmap: 2024-03-29T10:48:47.190 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:47.190 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:47.190 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:48:47.206736+0000 2024-03-29T10:48:47.190 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:48:47.206736+0000 2024-03-29T10:48:47.190 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:47.190 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:47.190 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:47.190 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:48:47.190 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:48:47.190 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:47.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.205+0000 7fe7857d6640 1 Processor -- start 2024-03-29T10:48:47.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.205+0000 7fe7857d6640 1 -- start start 2024-03-29T10:48:47.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.205+0000 7fe7857d6640 10 monclient: init 2024-03-29T10:48:47.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.206+0000 7fe7857d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:47.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.206+0000 7fe7857d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:47.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.206+0000 7fe7857d6640 10 monclient(hunting): picked mon.noname-b con 0x7fe7800a9930 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:47.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.206+0000 7fe7857d6640 10 monclient(hunting): picked mon.noname-a con 0x7fe780149dc0 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:47.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.206+0000 7fe7857d6640 10 monclient(hunting): picked mon.noname-c con 0x7fe78014ff40 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:47.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.206+0000 7fe7857d6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7800aae60 con 0x7fe780149dc0 2024-03-29T10:48:47.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.206+0000 7fe7857d6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7800b4310 con 0x7fe78014ff40 2024-03-29T10:48:47.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.206+0000 7fe7857d6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7800afdd0 con 0x7fe7800a9930 2024-03-29T10:48:47.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.206+0000 7fe7857d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:47.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.206+0000 7fe7857d6640 10 monclient(hunting): authenticate will time out at 1147.404833s 2024-03-29T10:48:47.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.210+0000 7fe77f7fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe78014ff40 0x7fe780152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:53543/0 (socket says 172.21.15.98:53543) 2024-03-29T10:48:47.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.210+0000 7fe77f7fe640 1 -- 172.21.15.98:0/797376611 learned_addr learned my addr 172.21.15.98:0/797376611 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:47.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.211+0000 7fe77dffb640 1 -- 172.21.15.98:0/797376611 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481802319 0 0) 0x7fe7800b4310 con 0x7fe78014ff40 2024-03-29T10:48:47.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.211+0000 7fe77dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:47.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.211+0000 7fe77dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:47.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.211+0000 7fe77dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:47.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.211+0000 7fe77dffb640 10 monclient(hunting): my global_id is 4372 2024-03-29T10:48:47.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.211+0000 7fe77dffb640 1 -- 172.21.15.98:0/797376611 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe768003520 con 0x7fe78014ff40 2024-03-29T10:48:47.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.212+0000 7fe77dffb640 1 -- 172.21.15.98:0/797376611 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4191567287 0 0) 0x7fe768003520 con 0x7fe78014ff40 2024-03-29T10:48:47.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.212+0000 7fe77dffb640 1 -- 172.21.15.98:0/797376611 >> v1:172.21.15.98:6790/0 conn(0x7fe7800a9930 legacy=0x7fe780143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:47.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.213+0000 7fe77dffb640 1 -- 172.21.15.98:0/797376611 >> v1:172.21.15.98:6789/0 conn(0x7fe780149dc0 legacy=0x7fe78014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:47.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.213+0000 7fe77dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:47.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.213+0000 7fe77dffb640 1 monclient: found mon.noname-c 2024-03-29T10:48:47.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.213+0000 7fe77dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:47.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.213+0000 7fe77dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:48:47.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.213+0000 7fe77dffb640 1 -- 172.21.15.98:0/797376611 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78014a950 con 0x7fe78014ff40 2024-03-29T10:48:47.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.214+0000 7fe77dffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:47.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.214+0000 7fe77dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:47.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.214+0000 7fe77dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:47.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.214+0000 7fe77dffb640 1 -- 172.21.15.98:0/797376611 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe7780019f0 con 0x7fe78014ff40 2024-03-29T10:48:47.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.214+0000 7fe77dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:47.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.214+0000 7fe77dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:47.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.214+0000 7fe77dffb640 10 monclient: dump: 2024-03-29T10:48:47.198 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:47.199 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:47.199 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:47.199 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:47.199 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:47.199 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:47.199 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:47.199 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:47.199 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:47.199 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:47.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.214+0000 7fe7857d6640 5 monclient: authenticate success, global_id 4372 2024-03-29T10:48:47.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.214+0000 7fe7857d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:47.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.214+0000 7fe77dffb640 1 -- 172.21.15.98:0/797376611 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe778001c80 con 0x7fe78014ff40 2024-03-29T10:48:47.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.214+0000 7fe77dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:47.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.214+0000 7fe7857d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:47.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.214+0000 7fe7857d6640 4 set_mon_vals no callback set 2024-03-29T10:48:47.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.215+0000 7fe7857d6640 10 monclient: shutdown 2024-03-29T10:48:47.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.215+0000 7fe7857d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:47.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.215+0000 7fe7857d6640 1 -- 172.21.15.98:0/797376611 >> v1:172.21.15.189:6789/0 conn(0x7fe78014ff40 legacy=0x7fe780152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:47.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.215+0000 7fe77dffb640 1 -- 172.21.15.98:0/797376611 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe778002050 con 0x7fe78014ff40 2024-03-29T10:48:47.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.216+0000 7fe77dffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:48:47.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.216+0000 7fe7857d6640 1 -- 172.21.15.98:0/797376611 shutdown_connections 2024-03-29T10:48:47.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.216+0000 7fe7857d6640 1 -- 172.21.15.98:0/797376611 >> 172.21.15.98:0/797376611 conn(0x7fe7800b35e0 msgr2=0x7fe7800b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:47.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.216+0000 7fe7857d6640 1 -- 172.21.15.98:0/797376611 shutdown_connections 2024-03-29T10:48:47.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.216+0000 7fe7857d6640 1 -- 172.21.15.98:0/797376611 wait complete. 2024-03-29T10:48:47.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.217+0000 7fe7857d6640 10 monclient: build_initial_monmap 2024-03-29T10:48:47.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.217+0000 7fe7857d6640 10 monclient: monmap: 2024-03-29T10:48:47.202 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:47.202 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:47.202 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:48:47.202 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:48:47.202 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:47.203 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:47.203 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:47.203 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:48:47.203 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:48:47.203 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:47.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.218+0000 7fe7857d6640 1 librados: starting msgr at 2024-03-29T10:48:47.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.218+0000 7fe7857d6640 1 librados: starting objecter 2024-03-29T10:48:47.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.219+0000 7fe7857d6640 1 Processor -- start 2024-03-29T10:48:47.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.220+0000 7fe7857d6640 1 -- start start 2024-03-29T10:48:47.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.220+0000 7fe7857d6640 1 librados: setting wanted keys 2024-03-29T10:48:47.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.220+0000 7fe7857d6640 1 librados: calling monclient init 2024-03-29T10:48:47.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.220+0000 7fe7857d6640 10 monclient: init 2024-03-29T10:48:47.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.220+0000 7fe7857d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:47.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.220+0000 7fe7857d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:47.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.220+0000 7fe7857d6640 10 monclient(hunting): picked mon.noname-b con 0x7fe7800a9930 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:47.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.227+0000 7fe77e7fc640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe780149dc0 0x7fe78013d280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57509/0 (socket says 172.21.15.98:57509) 2024-03-29T10:48:47.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.227+0000 7fe7857d6640 10 monclient(hunting): picked mon.noname-a con 0x7fe780149dc0 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:47.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.227+0000 7fe77effd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe7800a9930 0x7fe7801df240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:53649/0 (socket says 172.21.15.98:53649) 2024-03-29T10:48:47.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.228+0000 7fe7857d6640 10 monclient(hunting): picked mon.noname-c con 0x7fe78014ff40 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:47.214 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.228+0000 7fe7857d6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78011cef0 con 0x7fe780149dc0 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.228+0000 7fe7857d6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7801df950 con 0x7fe7800a9930 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.228+0000 7fe7857d6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7801ded20 con 0x7fe78014ff40 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.228+0000 7fe7857d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.228+0000 7fe7857d6640 10 monclient(hunting): authenticate will time out at 1147.426437s 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.228+0000 7fe77e7fc640 1 -- 172.21.15.98:0/3718064817 learned_addr learned my addr 172.21.15.98:0/3718064817 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.229+0000 7fe7677fe640 10 client.?.objecter ms_handle_connect 0x7fe780149dc0 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.229+0000 7fe7677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.231+0000 7fe7677fe640 10 client.?.objecter ms_handle_connect 0x7fe7800a9930 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.231+0000 7fe7677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.231+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380914398 0 0) 0x7fe78011cef0 con 0x7fe780149dc0 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.231+0000 7fe7677fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.231+0000 7fe7677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.231+0000 7fe7677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.231+0000 7fe7677fe640 10 monclient(hunting): my global_id is 4371 2024-03-29T10:48:47.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.231+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe76c004eb0 con 0x7fe780149dc0 2024-03-29T10:48:47.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.232+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952104046 0 0) 0x7fe7801df950 con 0x7fe7800a9930 2024-03-29T10:48:47.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.232+0000 7fe7677fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:47.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.232+0000 7fe7677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:47.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.232+0000 7fe7677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:47.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.232+0000 7fe7677fe640 10 monclient(hunting): my global_id is 4375 2024-03-29T10:48:47.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.232+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe78011cef0 con 0x7fe7800a9930 2024-03-29T10:48:47.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.232+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4057099829 0 0) 0x7fe76c004eb0 con 0x7fe780149dc0 2024-03-29T10:48:47.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.232+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7801df950 con 0x7fe780149dc0 2024-03-29T10:48:47.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.232+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe7700019d0 con 0x7fe780149dc0 2024-03-29T10:48:47.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.232+0000 7fe7677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:47.222 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 10 monclient(hunting): dump: 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4189483172 0 0) 0x7fe78011cef0 con 0x7fe7800a9930 2024-03-29T10:48:47.223 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe76c004eb0 con 0x7fe7800a9930 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe774001860 con 0x7fe7800a9930 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 10 monclient(hunting): dump: 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:47.224 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2398819807 0 0) 0x7fe76c004eb0 con 0x7fe7800a9930 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 >> v1:172.21.15.98:6790/0 conn(0x7fe78014ff40 legacy=0x7fe7801e6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 >> v1:172.21.15.98:6789/0 conn(0x7fe780149dc0 legacy=0x7fe78013d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 1 monclient: found mon.b 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78011b1c0 con 0x7fe7800a9930 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7857d6640 5 monclient: authenticate success, global_id 4375 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7857d6640 10 monclient: _renew_subs 2024-03-29T10:48:47.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7857d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:48:47.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.233+0000 7fe7857d6640 1 -- 172.21.15.98:0/3718064817 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe780072120 con 0x7fe7800a9930 2024-03-29T10:48:47.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.234+0000 7fe7857d6640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:47.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.234+0000 7fe7857d6640 10 monclient: _renew_subs 2024-03-29T10:48:47.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.234+0000 7fe7857d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:48:47.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.234+0000 7fe7857d6640 1 -- 172.21.15.98:0/3718064817 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe78006f7f0 con 0x7fe7800a9930 2024-03-29T10:48:47.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.234+0000 7fe7857d6640 1 librados: init done 2024-03-29T10:48:47.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.235+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe774001d70 con 0x7fe7800a9930 2024-03-29T10:48:47.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.235+0000 7fe7677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:47.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.235+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe774001860 con 0x7fe7800a9930 2024-03-29T10:48:47.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.235+0000 7fe7677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:47.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.235+0000 7fe7677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:47.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.235+0000 7fe7677fe640 10 monclient: dump: 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.235+0000 7fe77d7fa640 4 set_mon_vals no callback set 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.236+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fe774003730 con 0x7fe7800a9930 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.236+0000 7fe7857d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.236+0000 7fe7857d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:47.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.236+0000 7fe7857d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:48:47.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.236+0000 7fe7857d6640 1 -- 172.21.15.98:0/3718064817 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe760005080 con 0x7fe7800a9930 2024-03-29T10:48:47.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.237+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fe7740844d0 con 0x7fe7800a9930 2024-03-29T10:48:47.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.237+0000 7fe7677fe640 10 client.4375.objecter ms_dispatch 0x7fe780000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:47.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.237+0000 7fe7677fe640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:47.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.237+0000 7fe7677fe640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:47.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.237+0000 7fe7677fe640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-29T10:48:47.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.237+0000 7fe77e7fc640 10 monclient: get_auth_request con 0x7fe76c06e220 auth_method 0 2024-03-29T10:48:47.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.238+0000 7fe7677fe640 10 client.4375.objecter ms_handle_connect 0x7fe76c06e220 2024-03-29T10:48:47.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.244+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe77404fe10 con 0x7fe7800a9930 2024-03-29T10:48:47.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.244+0000 7fe7677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:47.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.244+0000 7fe7677fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:47.591 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.608+0000 7fe7857d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:47.591 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.608+0000 7fe7857d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:47.591 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.608+0000 7fe7857d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:48:47.591 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.608+0000 7fe7857d6640 1 -- 172.21.15.98:0/3718064817 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe760005330 con 0x7fe7800a9930 2024-03-29T10:48:47.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.609+0000 7fe7677fe640 1 -- 172.21.15.98:0/3718064817 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fe774051d40 con 0x7fe7800a9930 2024-03-29T10:48:47.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.609+0000 7fe7677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:47.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.609+0000 7fe7677fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:47.595 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:48:47.595 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:47.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.615+0000 7fe7857d6640 10 librados: watch_flush enter 2024-03-29T10:48:47.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.615+0000 7fe7857d6640 10 librados: watch_flush exit 2024-03-29T10:48:47.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.615+0000 7fe7857d6640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-29T10:48:47.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.615+0000 7fe7857d6640 10 client.4375.objecter successfully canceled tick 2024-03-29T10:48:47.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.615+0000 7fe7857d6640 1 -- 172.21.15.98:0/3718064817 >> v1:172.21.15.98:6816/1481799426 conn(0x7fe76c06e220 legacy=0x7fe76c0706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:47.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.615+0000 7fe7857d6640 10 monclient: shutdown 2024-03-29T10:48:47.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.615+0000 7fe7857d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:47.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.615+0000 7fe7857d6640 1 -- 172.21.15.98:0/3718064817 >> v1:172.21.15.189:6789/0 conn(0x7fe7800a9930 legacy=0x7fe7801df240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:47.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.616+0000 7fe7857d6640 1 -- 172.21.15.98:0/3718064817 shutdown_connections 2024-03-29T10:48:47.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.616+0000 7fe7857d6640 1 -- 172.21.15.98:0/3718064817 >> 172.21.15.98:0/3718064817 conn(0x7fe7800b35e0 msgr2=0x7fe7800b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:47.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.616+0000 7fe7857d6640 1 -- 172.21.15.98:0/3718064817 shutdown_connections 2024-03-29T10:48:47.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.616+0000 7fe7857d6640 1 -- 172.21.15.98:0/3718064817 wait complete. 2024-03-29T10:48:47.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:47.616+0000 7fe7857d6640 1 librados: shutdown 2024-03-29T10:48:47.616 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:54.423 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:54.578 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.595+0000 7f37ce784640 10 monclient: get_monmap_and_config 2024-03-29T10:48:54.579 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.595+0000 7f37ce784640 10 monclient: build_initial_monmap 2024-03-29T10:48:54.579 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.595+0000 7f37ce784640 10 monclient: monmap: 2024-03-29T10:48:54.579 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:54.579 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:54.579 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:48:54.597369+0000 2024-03-29T10:48:54.579 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:48:54.597369+0000 2024-03-29T10:48:54.579 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:54.579 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:54.579 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:54.579 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:48:54.580 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:48:54.580 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:54.584 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.597+0000 7f37ce784640 1 Processor -- start 2024-03-29T10:48:54.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.597+0000 7f37ce784640 1 -- start start 2024-03-29T10:48:54.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.597+0000 7f37ce784640 10 monclient: init 2024-03-29T10:48:54.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.598+0000 7f37ce784640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:54.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.598+0000 7f37ce784640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:54.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.598+0000 7f37ce784640 10 monclient(hunting): picked mon.noname-a con 0x7f37c80a9750 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:54.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.598+0000 7f37ce784640 10 monclient(hunting): picked mon.noname-b con 0x7f37c80add40 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:54.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.598+0000 7f37ce784640 10 monclient(hunting): picked mon.noname-c con 0x7f37c80aad20 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:54.585 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.598+0000 7f37ce784640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c8149870 con 0x7f37c80a9750 2024-03-29T10:48:54.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.598+0000 7f37ce784640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c80abf80 con 0x7f37c80aad20 2024-03-29T10:48:54.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.598+0000 7f37ce784640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c8149ac0 con 0x7f37c80add40 2024-03-29T10:48:54.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.598+0000 7f37ce784640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:54.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.598+0000 7f37ce784640 10 monclient(hunting): authenticate will time out at 1154.796621s 2024-03-29T10:48:54.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.601+0000 7f37ccf81640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f37c80add40 0x7f37c8143610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:41796/0 (socket says 172.21.15.98:41796) 2024-03-29T10:48:54.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.601+0000 7f37ccf81640 1 -- 172.21.15.98:0/53347756 learned_addr learned my addr 172.21.15.98:0/53347756 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:54.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.601+0000 7f37bffff640 1 -- 172.21.15.98:0/53347756 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577099010 0 0) 0x7f37c8149ac0 con 0x7f37c80add40 2024-03-29T10:48:54.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.601+0000 7f37bffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:54.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.601+0000 7f37bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:54.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.601+0000 7f37bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:54.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.601+0000 7f37bffff640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:48:54.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.601+0000 7f37bffff640 1 -- 172.21.15.98:0/53347756 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b4003520 con 0x7f37c80add40 2024-03-29T10:48:54.586 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.602+0000 7f37bffff640 1 -- 172.21.15.98:0/53347756 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2492139771 0 0) 0x7f37b4003520 con 0x7f37c80add40 2024-03-29T10:48:54.587 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.602+0000 7f37bffff640 1 -- 172.21.15.98:0/53347756 >> v1:172.21.15.189:6789/0 conn(0x7f37c80aad20 legacy=0x7f37c8151bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:54.591 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.604+0000 7f37bffff640 1 -- 172.21.15.98:0/53347756 >> v1:172.21.15.98:6789/0 conn(0x7f37c80a9750 legacy=0x7f37c8149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:54.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.604+0000 7f37bffff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:54.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.604+0000 7f37bffff640 1 monclient: found mon.noname-b 2024-03-29T10:48:54.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.604+0000 7f37bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:54.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.604+0000 7f37bffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:48:54.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.604+0000 7f37bffff640 1 -- 172.21.15.98:0/53347756 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c814a6f0 con 0x7f37c80add40 2024-03-29T10:48:54.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.605+0000 7f37bffff640 10 monclient: _finish_auth 0 2024-03-29T10:48:54.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.605+0000 7f37bffff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:54.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.605+0000 7f37bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:54.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.605+0000 7f37ce784640 5 monclient: authenticate success, global_id 4379 2024-03-29T10:48:54.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.605+0000 7f37ce784640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:54.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.605+0000 7f37bffff640 1 -- 172.21.15.98:0/53347756 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f37c0001a10 con 0x7f37c80add40 2024-03-29T10:48:54.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.605+0000 7f37bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:54.592 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.605+0000 7f37bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.605+0000 7f37bffff640 10 monclient: dump: 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.605+0000 7f37ce784640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.606+0000 7f37bffff640 1 -- 172.21.15.98:0/53347756 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37c0001ca0 con 0x7f37c80add40 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.606+0000 7f37bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:54.593 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.606+0000 7f37bffff640 1 -- 172.21.15.98:0/53347756 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f37c0002030 con 0x7f37c80add40 2024-03-29T10:48:54.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.606+0000 7f37bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:54.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.606+0000 7f37bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:54.594 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.606+0000 7f37bffff640 10 monclient: dump: 2024-03-29T10:48:54.594 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:54.594 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:54.594 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:54.594 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:54.594 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:54.594 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:54.594 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:54.594 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:54.594 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:54.594 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:54.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.606+0000 7f37ce784640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:54.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.606+0000 7f37ce784640 4 set_mon_vals no callback set 2024-03-29T10:48:54.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.607+0000 7f37ce784640 10 monclient: shutdown 2024-03-29T10:48:54.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.607+0000 7f37ce784640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:54.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.607+0000 7f37ce784640 1 -- 172.21.15.98:0/53347756 >> v1:172.21.15.98:6790/0 conn(0x7f37c80add40 legacy=0x7f37c8143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:54.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.608+0000 7f37ce784640 1 -- 172.21.15.98:0/53347756 shutdown_connections 2024-03-29T10:48:54.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.608+0000 7f37ce784640 1 -- 172.21.15.98:0/53347756 >> 172.21.15.98:0/53347756 conn(0x7f37c80673e0 msgr2=0x7f37c80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:54.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.608+0000 7f37ce784640 1 -- 172.21.15.98:0/53347756 shutdown_connections 2024-03-29T10:48:54.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.608+0000 7f37ce784640 1 -- 172.21.15.98:0/53347756 wait complete. 2024-03-29T10:48:54.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.609+0000 7f37ce784640 10 monclient: build_initial_monmap 2024-03-29T10:48:54.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.609+0000 7f37ce784640 10 monclient: monmap: 2024-03-29T10:48:54.597 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:48:54.597 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:54.597 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:48:54.597 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:48:54.597 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:54.597 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:54.597 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.610+0000 7f37ce784640 1 librados: starting msgr at 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.610+0000 7f37ce784640 1 librados: starting objecter 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.610+0000 7f37ce784640 1 Processor -- start 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.610+0000 7f37ce784640 1 -- start start 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.610+0000 7f37ce784640 1 librados: setting wanted keys 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.610+0000 7f37ce784640 1 librados: calling monclient init 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.610+0000 7f37ce784640 10 monclient: init 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.611+0000 7f37ce784640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.611+0000 7f37ce784640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.611+0000 7f37ce784640 10 monclient(hunting): picked mon.noname-b con 0x7f37c80a9750 addr v1:172.21.15.189:6789/0 2024-03-29T10:48:54.598 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.611+0000 7f37ce784640 10 monclient(hunting): picked mon.noname-a con 0x7f37c80aad20 addr v1:172.21.15.98:6789/0 2024-03-29T10:48:54.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.611+0000 7f37ce784640 10 monclient(hunting): picked mon.noname-c con 0x7f37c80add40 addr v1:172.21.15.98:6790/0 2024-03-29T10:48:54.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.611+0000 7f37ce784640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c8146da0 con 0x7f37c80aad20 2024-03-29T10:48:54.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.611+0000 7f37ce784640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c806e0d0 con 0x7f37c80a9750 2024-03-29T10:48:54.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.611+0000 7f37ce784640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c8146880 con 0x7f37c80add40 2024-03-29T10:48:54.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.611+0000 7f37ce784640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:54.599 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.611+0000 7f37ce784640 10 monclient(hunting): authenticate will time out at 1154.809286s 2024-03-29T10:48:54.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.615+0000 7f37ccf81640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f37c80aad20 0x7f37c81db2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54517/0 (socket says 172.21.15.98:54517) 2024-03-29T10:48:54.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.615+0000 7f37ccf81640 1 -- 172.21.15.98:0/1728098983 learned_addr learned my addr 172.21.15.98:0/1728098983 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:48:54.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.615+0000 7f37bdffb640 10 client.?.objecter ms_handle_connect 0x7f37c80add40 2024-03-29T10:48:54.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.615+0000 7f37bdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:54.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335036766 0 0) 0x7f37c8146880 con 0x7f37c80add40 2024-03-29T10:48:54.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:54.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:54.600 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 10 monclient(hunting): my global_id is 4382 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ac004eb0 con 0x7f37c80add40 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3398712151 0 0) 0x7f37ac004eb0 con 0x7f37c80add40 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37c8146880 con 0x7f37c80add40 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 10 client.?.objecter ms_handle_connect 0x7f37c80aad20 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f37b80017f0 con 0x7f37c80add40 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 10 monclient(hunting): dump: 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:54.601 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:54.602 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:54.602 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:54.602 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:54.602 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:54.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.616+0000 7f37bdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:54.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.617+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2940755589 0 0) 0x7f37c8146880 con 0x7f37c80add40 2024-03-29T10:48:54.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.617+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 >> v1:172.21.15.189:6789/0 conn(0x7f37c80a9750 legacy=0x7f37c813c0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:54.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 >> v1:172.21.15.98:6789/0 conn(0x7f37c80aad20 legacy=0x7f37c81db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:54.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37bdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:54.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37bdffb640 1 monclient: found mon.c 2024-03-29T10:48:54.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:54.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37bdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:54.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c8145c50 con 0x7f37c80add40 2024-03-29T10:48:54.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37bdffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:54.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37bdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:54.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:54.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37bdffb640 10 client.?.objecter ms_handle_connect 0x7f37c80a9750 2024-03-29T10:48:54.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37bdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:54.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37ce784640 5 monclient: authenticate success, global_id 4382 2024-03-29T10:48:54.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37ce784640 10 monclient: _renew_subs 2024-03-29T10:48:54.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37ce784640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:54.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37ce784640 1 -- 172.21.15.98:0/1728098983 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c8145ea0 con 0x7f37c80add40 2024-03-29T10:48:54.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37ce784640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:54.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37ce784640 10 monclient: _renew_subs 2024-03-29T10:48:54.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37ce784640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:54.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37ce784640 1 -- 172.21.15.98:0/1728098983 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c8146490 con 0x7f37c80add40 2024-03-29T10:48:54.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.620+0000 7f37ce784640 1 librados: init done 2024-03-29T10:48:54.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.621+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37b8001dc0 con 0x7f37c80add40 2024-03-29T10:48:54.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.622+0000 7f37bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:54.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.622+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f37b8003830 con 0x7f37c80add40 2024-03-29T10:48:54.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.622+0000 7f37bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:54.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.622+0000 7f37bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:54.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.622+0000 7f37bdffb640 10 monclient: dump: 2024-03-29T10:48:54.607 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:48:54.607 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:48:54.607 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:54.607 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:48:54.607 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:54.607 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:48:54.607 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:48:54.607 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:48:54.607 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:48:54.607 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:48:54.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.622+0000 7f37bf7fe640 4 set_mon_vals no callback set 2024-03-29T10:48:54.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.622+0000 7f37ce784640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:54.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.623+0000 7f37ce784640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:54.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.623+0000 7f37ce784640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:54.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.623+0000 7f37ce784640 1 -- 172.21.15.98:0/1728098983 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a8005080 con 0x7f37c80add40 2024-03-29T10:48:54.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.624+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f37b8019800 con 0x7f37c80add40 2024-03-29T10:48:54.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.626+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f37b8083800 con 0x7f37c80add40 2024-03-29T10:48:54.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.626+0000 7f37bdffb640 10 client.4382.objecter ms_dispatch 0x7f37c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:54.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.626+0000 7f37bdffb640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:54.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.626+0000 7f37bdffb640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:54.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.626+0000 7f37bdffb640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-29T10:48:54.611 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.629+0000 7f37cd782640 10 monclient: get_auth_request con 0x7f37ac06db80 auth_method 0 2024-03-29T10:48:54.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.630+0000 7f37bdffb640 10 client.4382.objecter ms_handle_connect 0x7f37ac06db80 2024-03-29T10:48:54.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.636+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37b804f140 con 0x7f37c80add40 2024-03-29T10:48:54.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.637+0000 7f37bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:54.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:54.637+0000 7f37bdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:54.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.000+0000 7f37ce784640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:54.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.000+0000 7f37ce784640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:54.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.000+0000 7f37ce784640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:48:54.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.000+0000 7f37ce784640 1 -- 172.21.15.98:0/1728098983 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37a8005330 con 0x7f37c80add40 2024-03-29T10:48:54.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.003+0000 7f37bdffb640 1 -- 172.21.15.98:0/1728098983 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f37b8051070 con 0x7f37c80add40 2024-03-29T10:48:54.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.003+0000 7f37bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:54.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.003+0000 7f37bdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:54.986 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:48:54.987 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:54.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.007+0000 7f37ce784640 10 librados: watch_flush enter 2024-03-29T10:48:54.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.008+0000 7f37ce784640 10 librados: watch_flush exit 2024-03-29T10:48:54.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.008+0000 7f37ce784640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-29T10:48:54.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.008+0000 7f37ce784640 10 client.4382.objecter successfully canceled tick 2024-03-29T10:48:54.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.009+0000 7f37ce784640 1 -- 172.21.15.98:0/1728098983 >> v1:172.21.15.98:6816/1481799426 conn(0x7f37ac06db80 legacy=0x7f37ac070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:54.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.009+0000 7f37ce784640 10 monclient: shutdown 2024-03-29T10:48:54.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.009+0000 7f37ce784640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:54.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.009+0000 7f37ce784640 1 -- 172.21.15.98:0/1728098983 >> v1:172.21.15.98:6790/0 conn(0x7f37c80add40 legacy=0x7f37c81e6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:54.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.009+0000 7f37ce784640 1 -- 172.21.15.98:0/1728098983 shutdown_connections 2024-03-29T10:48:54.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.009+0000 7f37ce784640 1 -- 172.21.15.98:0/1728098983 >> 172.21.15.98:0/1728098983 conn(0x7f37c80673e0 msgr2=0x7f37c81416d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:54.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.009+0000 7f37ce784640 1 -- 172.21.15.98:0/1728098983 shutdown_connections 2024-03-29T10:48:54.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.009+0000 7f37ce784640 1 -- 172.21.15.98:0/1728098983 wait complete. 2024-03-29T10:48:54.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:48:55.009+0000 7f37ce784640 1 librados: shutdown 2024-03-29T10:48:55.013 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:01.821 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:01.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.995+0000 7f865f59e640 10 monclient: get_monmap_and_config 2024-03-29T10:49:01.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.995+0000 7f865f59e640 10 monclient: build_initial_monmap 2024-03-29T10:49:01.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.995+0000 7f865f59e640 10 monclient: monmap: 2024-03-29T10:49:01.978 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:01.978 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:01.978 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:49:01.996545+0000 2024-03-29T10:49:01.978 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:49:01.996545+0000 2024-03-29T10:49:01.978 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:01.978 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:01.978 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:01.978 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:49:01.978 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:49:01.978 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:01.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.996+0000 7f865f59e640 1 Processor -- start 2024-03-29T10:49:01.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.996+0000 7f865f59e640 1 -- start start 2024-03-29T10:49:01.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.996+0000 7f865f59e640 10 monclient: init 2024-03-29T10:49:01.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.997+0000 7f865f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:01.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.997+0000 7f865f59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:01.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.997+0000 7f865f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f86580389e0 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:01.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8658038dc0 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:01.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f8658030660 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:01.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865f59e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86580c54c0 con 0x7f8658030660 2024-03-29T10:49:01.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865f59e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8658036430 con 0x7f8658038dc0 2024-03-29T10:49:01.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865f59e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8658039840 con 0x7f86580389e0 2024-03-29T10:49:01.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:01.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865f59e640 10 monclient(hunting): authenticate will time out at 1162.195029s 2024-03-29T10:49:01.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865e59c640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f86580389e0 0x7f8658033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:42696/0 (socket says 172.21.15.98:42696) 2024-03-29T10:49:01.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865e59c640 1 -- 172.21.15.98:0/3824243324 learned_addr learned my addr 172.21.15.98:0/3824243324 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:01.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865d59a640 1 -- 172.21.15.98:0/3824243324 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 913114859 0 0) 0x7f8658036430 con 0x7f8658038dc0 2024-03-29T10:49:01.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:01.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:01.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:01.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865d59a640 10 monclient(hunting): my global_id is 4381 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865d59a640 1 -- 172.21.15.98:0/3824243324 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8648003520 con 0x7f8658038dc0 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865d59a640 1 -- 172.21.15.98:0/3824243324 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209301083 0 0) 0x7f8658039840 con 0x7f86580389e0 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865d59a640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.998+0000 7f865d59a640 1 -- 172.21.15.98:0/3824243324 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8658036430 con 0x7f86580389e0 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 1 -- 172.21.15.98:0/3824243324 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764254373 0 0) 0x7f86580c54c0 con 0x7f8658030660 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 1 -- 172.21.15.98:0/3824243324 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8658039840 con 0x7f8658030660 2024-03-29T10:49:01.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 1 -- 172.21.15.98:0/3824243324 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1006335463 0 0) 0x7f8648003520 con 0x7f8658038dc0 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 1 -- 172.21.15.98:0/3824243324 >> v1:172.21.15.98:6790/0 conn(0x7f86580389e0 legacy=0x7f8658033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 1 -- 172.21.15.98:0/3824243324 >> v1:172.21.15.98:6789/0 conn(0x7f8658030660 legacy=0x7f86580d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 1 monclient: found mon.noname-c 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 1 -- 172.21.15.98:0/3824243324 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8658035690 con 0x7f8658038dc0 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 10 monclient: _finish_auth 0 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 1 -- 172.21.15.98:0/3824243324 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8654001690 con 0x7f8658038dc0 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:01.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865d59a640 10 monclient: dump: 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865f59e640 5 monclient: authenticate success, global_id 4381 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:01.999+0000 7f865f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.000+0000 7f865d59a640 1 -- 172.21.15.98:0/3824243324 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86540018b0 con 0x7f8658038dc0 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.000+0000 7f865d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:01.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.000+0000 7f865d59a640 1 -- 172.21.15.98:0/3824243324 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8654001c40 con 0x7f8658038dc0 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.000+0000 7f865f59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.000+0000 7f865f59e640 4 set_mon_vals no callback set 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.000+0000 7f865d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.000+0000 7f865d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.000+0000 7f865d59a640 10 monclient: dump: 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:01.988 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.001+0000 7f865f59e640 10 monclient: shutdown 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.001+0000 7f865f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.001+0000 7f865f59e640 1 -- 172.21.15.98:0/3824243324 >> v1:172.21.15.189:6789/0 conn(0x7f8658038dc0 legacy=0x7f86580cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.001+0000 7f865f59e640 1 -- 172.21.15.98:0/3824243324 shutdown_connections 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.001+0000 7f865f59e640 1 -- 172.21.15.98:0/3824243324 >> 172.21.15.98:0/3824243324 conn(0x7f8658005c00 msgr2=0x7f8658006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.001+0000 7f865f59e640 1 -- 172.21.15.98:0/3824243324 shutdown_connections 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.001+0000 7f865f59e640 1 -- 172.21.15.98:0/3824243324 wait complete. 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.002+0000 7f865f59e640 10 monclient: build_initial_monmap 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.002+0000 7f865f59e640 10 monclient: monmap: 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:49:01.990 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:01.990 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:01.990 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:01.990 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:49:01.990 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:49:01.990 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:01.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.003+0000 7f865f59e640 1 librados: starting msgr at 2024-03-29T10:49:01.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.003+0000 7f865f59e640 1 librados: starting objecter 2024-03-29T10:49:01.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.003+0000 7f865f59e640 1 Processor -- start 2024-03-29T10:49:01.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.003+0000 7f865f59e640 1 -- start start 2024-03-29T10:49:01.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.003+0000 7f865f59e640 1 librados: setting wanted keys 2024-03-29T10:49:01.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.003+0000 7f865f59e640 1 librados: calling monclient init 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.003+0000 7f865f59e640 10 monclient: init 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f865f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f865f59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f865f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8658030660 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f865f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f86580389e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f865f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f865802eaf0 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f865f59e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86580c7a40 con 0x7f865802eaf0 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f865f59e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86580c7bd0 con 0x7f86580389e0 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f865f59e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86580c7d60 con 0x7f8658030660 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f865f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f865f59e640 10 monclient(hunting): authenticate will time out at 1162.201186s 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f865e59c640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f8658030660 0x7f86580c29b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:42910/0 (socket says 172.21.15.98:42910) 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f865e59c640 1 -- 172.21.15.98:0/4114494877 learned_addr learned my addr 172.21.15.98:0/4114494877 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f864effd640 10 client.?.objecter ms_handle_connect 0x7f8658030660 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f864effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f864effd640 10 client.?.objecter ms_handle_connect 0x7f865802eaf0 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f864effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842060318 0 0) 0x7f86580c7d60 con 0x7f8658030660 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f864effd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f864effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.004+0000 7f864effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8640004eb0 con 0x7f8658030660 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 client.?.objecter ms_handle_connect 0x7f86580389e0 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548198531 0 0) 0x7f86580c7bd0 con 0x7f86580389e0 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:01.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): my global_id is 4384 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86580c7d60 con 0x7f86580389e0 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441973176 0 0) 0x7f86580c7a40 con 0x7f865802eaf0 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): my global_id is 4383 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86580c7bd0 con 0x7f865802eaf0 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4264402782 0 0) 0x7f8640004eb0 con 0x7f8658030660 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86580c7a40 con 0x7f8658030660 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8650001930 con 0x7f8658030660 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:01.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): dump: 2024-03-29T10:49:01.999 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:01.999 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:01.999 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:01.999 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:01.999 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:01.999 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:01.999 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:01.999 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:01.999 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:01.999 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:01.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:01.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1433628738 0 0) 0x7f86580c7d60 con 0x7f86580389e0 2024-03-29T10:49:02.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8640004eb0 con 0x7f86580389e0 2024-03-29T10:49:02.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8654001bc0 con 0x7f86580389e0 2024-03-29T10:49:02.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:02.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.005+0000 7f864effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:02.000 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 10 monclient(hunting): dump: 2024-03-29T10:49:02.000 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:02.000 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:02.000 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:02.000 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:02.000 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:02.000 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:02.000 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2814926600 0 0) 0x7f86580c7bd0 con 0x7f865802eaf0 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86580c7d60 con 0x7f865802eaf0 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f866004fdd0 con 0x7f865802eaf0 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 10 monclient(hunting): dump: 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:02.001 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:02.002 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:02.002 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:02.002 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:02.002 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:02.002 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:02.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4066986836 0 0) 0x7f86580c7a40 con 0x7f8658030660 2024-03-29T10:49:02.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 >> v1:172.21.15.189:6789/0 conn(0x7f86580389e0 legacy=0x7f8658160af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:02.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 >> v1:172.21.15.98:6789/0 conn(0x7f865802eaf0 legacy=0x7f8658161200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:02.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:02.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 1 monclient: found mon.c 2024-03-29T10:49:02.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:02.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:02.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86580c30c0 con 0x7f8658030660 2024-03-29T10:49:02.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 10 monclient: _finish_auth 0 2024-03-29T10:49:02.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:02.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:02.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f865f59e640 5 monclient: authenticate success, global_id 4388 2024-03-29T10:49:02.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f865f59e640 10 monclient: _renew_subs 2024-03-29T10:49:02.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f865f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:02.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f865f59e640 1 -- 172.21.15.98:0/4114494877 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86580c3310 con 0x7f8658030660 2024-03-29T10:49:02.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f865f59e640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:02.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f865f59e640 10 monclient: _renew_subs 2024-03-29T10:49:02.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f865f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f865f59e640 1 -- 172.21.15.98:0/4114494877 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f865802f770 con 0x7f8658030660 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f865f59e640 1 librados: init done 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8650001ab0 con 0x7f8658030660 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8650004a40 con 0x7f8658030660 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.006+0000 7f864effd640 10 monclient: dump: 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:02.004 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.007+0000 7f865cd99640 4 set_mon_vals no callback set 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.009+0000 7f865f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.009+0000 7f865f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.009+0000 7f865f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.009+0000 7f865f59e640 1 -- 172.21.15.98:0/4114494877 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8644005080 con 0x7f8658030660 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.010+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f8650004ba0 con 0x7f8658030660 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.011+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f8650084b60 con 0x7f8658030660 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.011+0000 7f865dd9b640 10 monclient: get_auth_request con 0x7f864006df70 auth_method 0 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.011+0000 7f864effd640 10 client.4388.objecter ms_dispatch 0x7f8660000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.011+0000 7f864effd640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:02.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.011+0000 7f864effd640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:02.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.011+0000 7f864effd640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-29T10:49:02.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.011+0000 7f864effd640 10 client.4388.objecter ms_handle_connect 0x7f864006df70 2024-03-29T10:49:02.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.020+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8650050270 con 0x7f8658030660 2024-03-29T10:49:02.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.020+0000 7f864effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:02.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.020+0000 7f864effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:02.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.408+0000 7f865f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:02.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.408+0000 7f865f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:02.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.408+0000 7f865f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:02.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.408+0000 7f865f59e640 1 -- 172.21.15.98:0/4114494877 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8644005330 con 0x7f8658030660 2024-03-29T10:49:02.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.409+0000 7f864effd640 1 -- 172.21.15.98:0/4114494877 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f8650052370 con 0x7f8658030660 2024-03-29T10:49:02.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.409+0000 7f864effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:02.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.409+0000 7f864effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:02.391 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:49:02.391 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:02.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.412+0000 7f865f59e640 10 librados: watch_flush enter 2024-03-29T10:49:02.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.412+0000 7f865f59e640 10 librados: watch_flush exit 2024-03-29T10:49:02.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.412+0000 7f865f59e640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-29T10:49:02.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.412+0000 7f865f59e640 10 client.4388.objecter successfully canceled tick 2024-03-29T10:49:02.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.413+0000 7f865f59e640 1 -- 172.21.15.98:0/4114494877 >> v1:172.21.15.98:6816/1481799426 conn(0x7f864006df70 legacy=0x7f8640070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:02.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.413+0000 7f865f59e640 10 monclient: shutdown 2024-03-29T10:49:02.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.413+0000 7f865f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:02.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.413+0000 7f865f59e640 1 -- 172.21.15.98:0/4114494877 >> v1:172.21.15.98:6790/0 conn(0x7f8658030660 legacy=0x7f86580c29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:02.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.413+0000 7f865f59e640 1 -- 172.21.15.98:0/4114494877 shutdown_connections 2024-03-29T10:49:02.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.413+0000 7f865f59e640 1 -- 172.21.15.98:0/4114494877 >> 172.21.15.98:0/4114494877 conn(0x7f8658005c00 msgr2=0x7f8658005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:02.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.413+0000 7f865f59e640 1 -- 172.21.15.98:0/4114494877 shutdown_connections 2024-03-29T10:49:02.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.413+0000 7f865f59e640 1 -- 172.21.15.98:0/4114494877 wait complete. 2024-03-29T10:49:02.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:02.413+0000 7f865f59e640 1 librados: shutdown 2024-03-29T10:49:02.410 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:09.218 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:09.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.385+0000 7ff6521f4640 10 monclient: get_monmap_and_config 2024-03-29T10:49:09.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.385+0000 7ff6521f4640 10 monclient: build_initial_monmap 2024-03-29T10:49:09.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.385+0000 7ff6521f4640 10 monclient: monmap: 2024-03-29T10:49:09.368 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:09.368 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:09.368 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:49:09.386466+0000 2024-03-29T10:49:09.368 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:49:09.386466+0000 2024-03-29T10:49:09.368 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:09.368 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:09.368 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:09.368 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:49:09.368 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:49:09.368 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:09.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.386+0000 7ff6521f4640 1 Processor -- start 2024-03-29T10:49:09.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.387+0000 7ff6521f4640 1 -- start start 2024-03-29T10:49:09.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.387+0000 7ff6521f4640 10 monclient: init 2024-03-29T10:49:09.369 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.388+0000 7ff6521f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:09.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.388+0000 7ff6521f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:09.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.388+0000 7ff6521f4640 10 monclient(hunting): picked mon.noname-b con 0x7ff64c1457f0 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:09.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.388+0000 7ff6521f4640 10 monclient(hunting): picked mon.noname-a con 0x7ff64c13f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:09.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.388+0000 7ff6521f4640 10 monclient(hunting): picked mon.noname-c con 0x7ff64c0a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:09.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.388+0000 7ff6521f4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c0aaef0 con 0x7ff64c13f650 2024-03-29T10:49:09.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.388+0000 7ff6521f4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c145bd0 con 0x7ff64c0a9d70 2024-03-29T10:49:09.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.388+0000 7ff6521f4640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c145eb0 con 0x7ff64c1457f0 2024-03-29T10:49:09.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.388+0000 7ff6521f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:09.370 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.388+0000 7ff6521f4640 10 monclient(hunting): authenticate will time out at 1169.585112s 2024-03-29T10:49:09.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.388+0000 7ff64bfff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff64c0a9d70 0x7ff64c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:44098/0 (socket says 172.21.15.98:44098) 2024-03-29T10:49:09.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.388+0000 7ff64bfff640 1 -- 172.21.15.98:0/3999941804 learned_addr learned my addr 172.21.15.98:0/3999941804 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:09.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 1 -- 172.21.15.98:0/3999941804 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844566120 0 0) 0x7ff64c0aaef0 con 0x7ff64c13f650 2024-03-29T10:49:09.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:09.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:09.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:09.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:49:09.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 1 -- 172.21.15.98:0/3999941804 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff638003520 con 0x7ff64c13f650 2024-03-29T10:49:09.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 1 -- 172.21.15.98:0/3999941804 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874593578 0 0) 0x7ff64c145eb0 con 0x7ff64c1457f0 2024-03-29T10:49:09.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:09.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:09.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:09.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:49:09.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 1 -- 172.21.15.98:0/3999941804 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff64c0aaef0 con 0x7ff64c1457f0 2024-03-29T10:49:09.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 1 -- 172.21.15.98:0/3999941804 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1681036411 0 0) 0x7ff638003520 con 0x7ff64c13f650 2024-03-29T10:49:09.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 1 -- 172.21.15.98:0/3999941804 >> v1:172.21.15.98:6790/0 conn(0x7ff64c1457f0 legacy=0x7ff64c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:09.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 1 -- 172.21.15.98:0/3999941804 >> v1:172.21.15.189:6789/0 conn(0x7ff64c0a9d70 legacy=0x7ff64c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:09.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:49:09.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 1 monclient: found mon.noname-a 2024-03-29T10:49:09.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:09.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:49:09.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 1 -- 172.21.15.98:0/3999941804 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c146340 con 0x7ff64c13f650 2024-03-29T10:49:09.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:49:09.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:49:09.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:09.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 1 -- 172.21.15.98:0/3999941804 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff644001690 con 0x7ff64c13f650 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.389+0000 7ff64a7fc640 10 monclient: dump: 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.390+0000 7ff6521f4640 5 monclient: authenticate success, global_id 4386 2024-03-29T10:49:09.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.390+0000 7ff6521f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:09.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.390+0000 7ff64a7fc640 1 -- 172.21.15.98:0/3999941804 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6440018b0 con 0x7ff64c13f650 2024-03-29T10:49:09.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.390+0000 7ff64a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:09.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.390+0000 7ff6521f4640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:09.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.390+0000 7ff6521f4640 4 set_mon_vals no callback set 2024-03-29T10:49:09.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.391+0000 7ff6521f4640 10 monclient: shutdown 2024-03-29T10:49:09.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.391+0000 7ff6521f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:09.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.391+0000 7ff6521f4640 1 -- 172.21.15.98:0/3999941804 >> v1:172.21.15.98:6789/0 conn(0x7ff64c13f650 legacy=0x7ff64c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.391+0000 7ff6521f4640 1 -- 172.21.15.98:0/3999941804 shutdown_connections 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.391+0000 7ff6521f4640 1 -- 172.21.15.98:0/3999941804 >> 172.21.15.98:0/3999941804 conn(0x7ff64c066a00 msgr2=0x7ff64c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.391+0000 7ff6521f4640 1 -- 172.21.15.98:0/3999941804 shutdown_connections 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.391+0000 7ff6521f4640 1 -- 172.21.15.98:0/3999941804 wait complete. 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.392+0000 7ff6521f4640 10 monclient: build_initial_monmap 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.392+0000 7ff6521f4640 10 monclient: monmap: 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:09.378 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.392+0000 7ff6521f4640 1 librados: starting msgr at 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.392+0000 7ff6521f4640 1 librados: starting objecter 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.392+0000 7ff6521f4640 1 Processor -- start 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.392+0000 7ff6521f4640 1 -- start start 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.392+0000 7ff6521f4640 1 librados: setting wanted keys 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.392+0000 7ff6521f4640 1 librados: calling monclient init 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.392+0000 7ff6521f4640 10 monclient: init 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.393+0000 7ff6521f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.393+0000 7ff6521f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.393+0000 7ff6521f4640 10 monclient(hunting): picked mon.noname-c con 0x7ff64c0a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.393+0000 7ff6521f4640 10 monclient(hunting): picked mon.noname-a con 0x7ff64c13f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:09.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.393+0000 7ff6521f4640 10 monclient(hunting): picked mon.noname-b con 0x7ff64c1457f0 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.393+0000 7ff6521f4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c0acdb0 con 0x7ff64c13f650 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.393+0000 7ff6521f4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c0ad090 con 0x7ff64c1457f0 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.393+0000 7ff6521f4640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c0ad370 con 0x7ff64c0a9d70 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.393+0000 7ff6521f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.393+0000 7ff6521f4640 10 monclient(hunting): authenticate will time out at 1169.590603s 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.393+0000 7ff64affd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff64c13f650 0x7ff64c1d7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41510/0 (socket says 172.21.15.98:41510) 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.393+0000 7ff64affd640 1 -- 172.21.15.98:0/2708285917 learned_addr learned my addr 172.21.15.98:0/2708285917 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 client.?.objecter ms_handle_connect 0x7ff64c13f650 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 client.?.objecter ms_handle_connect 0x7ff64c0a9d70 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869500140 0 0) 0x7ff64c0acdb0 con 0x7ff64c13f650 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:09.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 monclient(hunting): my global_id is 4389 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff62c004eb0 con 0x7ff64c13f650 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756460764 0 0) 0x7ff64c0ad370 con 0x7ff64c0a9d70 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 monclient(hunting): my global_id is 4394 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff64c0acdb0 con 0x7ff64c0a9d70 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1110049914 0 0) 0x7ff62c004eb0 con 0x7ff64c13f650 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff64c0ad370 con 0x7ff64c13f650 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff644003790 con 0x7ff64c13f650 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:09.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 monclient(hunting): dump: 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2692013549 0 0) 0x7ff64c0acdb0 con 0x7ff64c0a9d70 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.394+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff62c004eb0 con 0x7ff64c0a9d70 2024-03-29T10:49:09.382 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff634001890 con 0x7ff64c0a9d70 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 10 monclient(hunting): dump: 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 10 client.?.objecter ms_handle_connect 0x7ff64c1457f0 2024-03-29T10:49:09.383 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:09.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4009855607 0 0) 0x7ff64c0ad370 con 0x7ff64c13f650 2024-03-29T10:49:09.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 >> v1:172.21.15.98:6790/0 conn(0x7ff64c0a9d70 legacy=0x7ff64c137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:09.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 >> v1:172.21.15.189:6789/0 conn(0x7ff64c1457f0 legacy=0x7ff64c1e29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:09.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:09.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 1 monclient: found mon.a 2024-03-29T10:49:09.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:09.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:49:09.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c0abac0 con 0x7ff64c13f650 2024-03-29T10:49:09.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 10 monclient: _finish_auth 0 2024-03-29T10:49:09.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:09.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff643fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:09.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff6521f4640 5 monclient: authenticate success, global_id 4389 2024-03-29T10:49:09.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff6521f4640 10 monclient: _renew_subs 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff6521f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff6521f4640 1 -- 172.21.15.98:0/2708285917 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64c0abd70 con 0x7ff64c13f650 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff6521f4640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff6521f4640 10 monclient: _renew_subs 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff6521f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff6521f4640 1 -- 172.21.15.98:0/2708285917 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64c13b9b0 con 0x7ff64c13f650 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.395+0000 7ff6521f4640 1 librados: init done 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.396+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff644001aa0 con 0x7ff64c13f650 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.396+0000 7ff643fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.396+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7ff644003e80 con 0x7ff64c13f650 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.396+0000 7ff643fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.396+0000 7ff643fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.396+0000 7ff643fff640 10 monclient: dump: 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:09.385 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:09.386 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:09.386 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:09.386 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:09.386 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:09.386 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:09.386 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:09.386 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:09.386 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:09.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.396+0000 7ff649ffb640 4 set_mon_vals no callback set 2024-03-29T10:49:09.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.396+0000 7ff6521f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:09.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.397+0000 7ff6521f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:09.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.397+0000 7ff6521f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:49:09.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.397+0000 7ff6521f4640 1 -- 172.21.15.98:0/2708285917 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff630005080 con 0x7ff64c13f650 2024-03-29T10:49:09.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.398+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7ff644003fe0 con 0x7ff64c13f650 2024-03-29T10:49:09.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.399+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7ff644084bc0 con 0x7ff64c13f650 2024-03-29T10:49:09.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.399+0000 7ff643fff640 10 client.4389.objecter ms_dispatch 0x7ff64c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:09.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.399+0000 7ff643fff640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:09.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.399+0000 7ff643fff640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:09.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.399+0000 7ff643fff640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-29T10:49:09.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.399+0000 7ff64b7fe640 10 monclient: get_auth_request con 0x7ff62c06dee0 auth_method 0 2024-03-29T10:49:09.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.400+0000 7ff643fff640 10 client.4389.objecter ms_handle_connect 0x7ff62c06dee0 2024-03-29T10:49:09.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.410+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6440504a0 con 0x7ff64c13f650 2024-03-29T10:49:09.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.410+0000 7ff643fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:09.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.410+0000 7ff643fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:09.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.781+0000 7ff6521f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:09.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.781+0000 7ff6521f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:09.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.781+0000 7ff6521f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:49:09.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.781+0000 7ff6521f4640 1 -- 172.21.15.98:0/2708285917 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff630005330 con 0x7ff64c13f650 2024-03-29T10:49:09.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.783+0000 7ff643fff640 1 -- 172.21.15.98:0/2708285917 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7ff6440523d0 con 0x7ff64c13f650 2024-03-29T10:49:09.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.783+0000 7ff643fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:09.765 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.783+0000 7ff643fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:09.765 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:49:09.765 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:09.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.786+0000 7ff6521f4640 10 librados: watch_flush enter 2024-03-29T10:49:09.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.786+0000 7ff6521f4640 10 librados: watch_flush exit 2024-03-29T10:49:09.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.786+0000 7ff6521f4640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-29T10:49:09.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.786+0000 7ff6521f4640 10 client.4389.objecter successfully canceled tick 2024-03-29T10:49:09.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.786+0000 7ff6521f4640 1 -- 172.21.15.98:0/2708285917 >> v1:172.21.15.98:6816/1481799426 conn(0x7ff62c06dee0 legacy=0x7ff62c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:09.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.786+0000 7ff6521f4640 10 monclient: shutdown 2024-03-29T10:49:09.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.786+0000 7ff6521f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:09.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.786+0000 7ff6521f4640 1 -- 172.21.15.98:0/2708285917 >> v1:172.21.15.98:6789/0 conn(0x7ff64c13f650 legacy=0x7ff64c1d7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:09.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.786+0000 7ff6521f4640 1 -- 172.21.15.98:0/2708285917 shutdown_connections 2024-03-29T10:49:09.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.787+0000 7ff6521f4640 1 -- 172.21.15.98:0/2708285917 >> 172.21.15.98:0/2708285917 conn(0x7ff64c066a00 msgr2=0x7ff64c13cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:09.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.787+0000 7ff6521f4640 1 -- 172.21.15.98:0/2708285917 shutdown_connections 2024-03-29T10:49:09.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.787+0000 7ff6521f4640 1 -- 172.21.15.98:0/2708285917 wait complete. 2024-03-29T10:49:09.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:09.787+0000 7ff6521f4640 1 librados: shutdown 2024-03-29T10:49:09.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:16.591 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:16.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.763+0000 7f5eb2154640 10 monclient: get_monmap_and_config 2024-03-29T10:49:16.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.763+0000 7f5eb2154640 10 monclient: build_initial_monmap 2024-03-29T10:49:16.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.763+0000 7f5eb2154640 10 monclient: monmap: 2024-03-29T10:49:16.745 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:16.745 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:16.745 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:49:16.764161+0000 2024-03-29T10:49:16.745 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:49:16.764161+0000 2024-03-29T10:49:16.745 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:16.745 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:16.746 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:16.746 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:49:16.746 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:49:16.746 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:16.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.764+0000 7f5eb2154640 1 Processor -- start 2024-03-29T10:49:16.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.764+0000 7f5eb2154640 1 -- start start 2024-03-29T10:49:16.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.764+0000 7f5eb2154640 10 monclient: init 2024-03-29T10:49:16.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.765+0000 7f5eb2154640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:16.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.765+0000 7f5eb2154640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:16.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.765+0000 7f5eb2154640 10 monclient(hunting): picked mon.noname-b con 0x7f5eac0b4330 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:16.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.765+0000 7f5eb2154640 10 monclient(hunting): picked mon.noname-c con 0x7f5eac1470b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:16.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.765+0000 7f5eb2154640 10 monclient(hunting): picked mon.noname-a con 0x7f5eac1439b0 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:16.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.765+0000 7f5eb2154640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eac06f2a0 con 0x7f5eac1439b0 2024-03-29T10:49:16.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.765+0000 7f5eb2154640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eac146b90 con 0x7f5eac1470b0 2024-03-29T10:49:16.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.765+0000 7f5eb2154640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eac0b2ca0 con 0x7f5eac0b4330 2024-03-29T10:49:16.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.765+0000 7f5eb2154640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:16.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.765+0000 7f5eb2154640 10 monclient(hunting): authenticate will time out at 1176.962614s 2024-03-29T10:49:16.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.772+0000 7f5eabfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5eac1439b0 0x7f5eac151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53185/0 (socket says 172.21.15.98:53185) 2024-03-29T10:49:16.754 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.772+0000 7f5eabfff640 1 -- 172.21.15.98:0/2922374553 learned_addr learned my addr 172.21.15.98:0/2922374553 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:16.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 1 -- 172.21.15.98:0/2922374553 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693140423 0 0) 0x7f5eac06f2a0 con 0x7f5eac1439b0 2024-03-29T10:49:16.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:16.757 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:16.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:16.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 10 monclient(hunting): my global_id is 4392 2024-03-29T10:49:16.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 1 -- 172.21.15.98:0/2922374553 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e94003520 con 0x7f5eac1439b0 2024-03-29T10:49:16.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 1 -- 172.21.15.98:0/2922374553 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733585841 0 0) 0x7f5eac0b2ca0 con 0x7f5eac0b4330 2024-03-29T10:49:16.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:16.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:16.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:16.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:49:16.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 1 -- 172.21.15.98:0/2922374553 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eac06f2a0 con 0x7f5eac0b4330 2024-03-29T10:49:16.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 1 -- 172.21.15.98:0/2922374553 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255709525 0 0) 0x7f5eac146b90 con 0x7f5eac1470b0 2024-03-29T10:49:16.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:16.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:16.758 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:16.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 10 monclient(hunting): my global_id is 4387 2024-03-29T10:49:16.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.774+0000 7f5eaa7fc640 1 -- 172.21.15.98:0/2922374553 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eac0b2ca0 con 0x7f5eac1470b0 2024-03-29T10:49:16.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 1 -- 172.21.15.98:0/2922374553 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3280054598 0 0) 0x7f5eac06f2a0 con 0x7f5eac0b4330 2024-03-29T10:49:16.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 1 -- 172.21.15.98:0/2922374553 >> v1:172.21.15.189:6789/0 conn(0x7f5eac1470b0 legacy=0x7f5eac0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:16.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 1 -- 172.21.15.98:0/2922374553 >> v1:172.21.15.98:6789/0 conn(0x7f5eac1439b0 legacy=0x7f5eac151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:16.759 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 1 monclient: found mon.noname-b 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 1 -- 172.21.15.98:0/2922374553 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac06e0c0 con 0x7f5eac0b4330 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 10 monclient: _finish_auth 0 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 1 -- 172.21.15.98:0/2922374553 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5e9c0015a0 con 0x7f5eac0b4330 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eaa7fc640 10 monclient: dump: 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:16.760 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eb2154640 5 monclient: authenticate success, global_id 4397 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.777+0000 7f5eb2154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.778+0000 7f5eaa7fc640 1 -- 172.21.15.98:0/2922374553 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e9c001700 con 0x7f5eac0b4330 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.778+0000 7f5eaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.778+0000 7f5eaa7fc640 1 -- 172.21.15.98:0/2922374553 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5e9c001b50 con 0x7f5eac0b4330 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.778+0000 7f5eaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:16.761 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.778+0000 7f5eaa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:16.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.778+0000 7f5eaa7fc640 10 monclient: dump: 2024-03-29T10:49:16.762 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:16.762 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:16.762 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:16.762 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:16.762 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:16.762 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:16.762 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:16.762 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:16.762 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:16.762 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:16.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.778+0000 7f5eb2154640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:16.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.778+0000 7f5eb2154640 4 set_mon_vals no callback set 2024-03-29T10:49:16.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.778+0000 7f5eb2154640 10 monclient: shutdown 2024-03-29T10:49:16.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.778+0000 7f5eb2154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:16.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.778+0000 7f5eb2154640 1 -- 172.21.15.98:0/2922374553 >> v1:172.21.15.98:6790/0 conn(0x7f5eac0b4330 legacy=0x7f5eac0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:16.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.779+0000 7f5eb2154640 1 -- 172.21.15.98:0/2922374553 shutdown_connections 2024-03-29T10:49:16.764 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.779+0000 7f5eb2154640 1 -- 172.21.15.98:0/2922374553 >> 172.21.15.98:0/2922374553 conn(0x7f5eac066a00 msgr2=0x7f5eac066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:16.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.780+0000 7f5eb2154640 1 -- 172.21.15.98:0/2922374553 shutdown_connections 2024-03-29T10:49:16.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.780+0000 7f5eb2154640 1 -- 172.21.15.98:0/2922374553 wait complete. 2024-03-29T10:49:16.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.780+0000 7f5eb2154640 10 monclient: build_initial_monmap 2024-03-29T10:49:16.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.780+0000 7f5eb2154640 10 monclient: monmap: 2024-03-29T10:49:16.766 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:16.766 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:16.766 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:49:16.766 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:49:16.766 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.781+0000 7f5eb2154640 1 librados: starting msgr at 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.781+0000 7f5eb2154640 1 librados: starting objecter 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.781+0000 7f5eb2154640 1 Processor -- start 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.781+0000 7f5eb2154640 1 -- start start 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.781+0000 7f5eb2154640 1 librados: setting wanted keys 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.781+0000 7f5eb2154640 1 librados: calling monclient init 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.781+0000 7f5eb2154640 10 monclient: init 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.782+0000 7f5eb2154640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.782+0000 7f5eb2154640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:16.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.782+0000 7f5eb2154640 10 monclient(hunting): picked mon.noname-c con 0x7f5eac0b4330 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:16.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.782+0000 7f5eb2154640 10 monclient(hunting): picked mon.noname-b con 0x7f5eac1439b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:16.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.782+0000 7f5eb2154640 10 monclient(hunting): picked mon.noname-a con 0x7f5eac1470b0 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:16.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.782+0000 7f5eb2154640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eac0ad7e0 con 0x7f5eac1470b0 2024-03-29T10:49:16.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.782+0000 7f5eb2154640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eac0ac070 con 0x7f5eac1439b0 2024-03-29T10:49:16.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.782+0000 7f5eb2154640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eac0abcb0 con 0x7f5eac0b4330 2024-03-29T10:49:16.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.782+0000 7f5eb2154640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:16.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.782+0000 7f5eb2154640 10 monclient(hunting): authenticate will time out at 1176.979554s 2024-03-29T10:49:16.770 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.788+0000 7f5eab7fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f5eac0b4330 0x7f5eac142880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:52013/0 (socket says 172.21.15.98:52013) 2024-03-29T10:49:16.770 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.788+0000 7f5eab7fe640 1 -- 172.21.15.98:0/2548582520 learned_addr learned my addr 172.21.15.98:0/2548582520 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:16.771 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.789+0000 7f5e93fff640 10 client.?.objecter ms_handle_connect 0x7f5eac0b4330 2024-03-29T10:49:16.771 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.789+0000 7f5e93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:16.771 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.789+0000 7f5e93fff640 10 client.?.objecter ms_handle_connect 0x7f5eac1439b0 2024-03-29T10:49:16.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.789+0000 7f5e93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:16.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.789+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385894478 0 0) 0x7f5eac0abcb0 con 0x7f5eac0b4330 2024-03-29T10:49:16.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.790+0000 7f5e93fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:16.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.790+0000 7f5e93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:16.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.790+0000 7f5e93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:16.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.790+0000 7f5e93fff640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:49:16.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.790+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e98004eb0 con 0x7f5eac0b4330 2024-03-29T10:49:16.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.790+0000 7f5e93fff640 10 client.?.objecter ms_handle_connect 0x7f5eac1470b0 2024-03-29T10:49:16.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.790+0000 7f5e93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:16.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.790+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953756796 0 0) 0x7f5eac0ac070 con 0x7f5eac1439b0 2024-03-29T10:49:16.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.790+0000 7f5e93fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:16.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.790+0000 7f5e93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:16.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.790+0000 7f5e93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:16.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.790+0000 7f5e93fff640 10 monclient(hunting): my global_id is 4390 2024-03-29T10:49:16.773 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.790+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eac0abcb0 con 0x7f5eac1439b0 2024-03-29T10:49:16.775 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.793+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2284109698 0 0) 0x7f5eac0abcb0 con 0x7f5eac1439b0 2024-03-29T10:49:16.775 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.793+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5eac0ac070 con 0x7f5eac1439b0 2024-03-29T10:49:16.775 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.793+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5ea40019f0 con 0x7f5eac1439b0 2024-03-29T10:49:16.775 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.793+0000 7f5e93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:16.775 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.793+0000 7f5e93fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:16.775 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.793+0000 7f5e93fff640 10 monclient(hunting): dump: 2024-03-29T10:49:16.775 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:16.775 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:16.775 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:16.775 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.793+0000 7f5e93fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.793+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1069366951 0 0) 0x7f5e98004eb0 con 0x7f5eac0b4330 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.793+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5eac0abcb0 con 0x7f5eac0b4330 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.793+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5e9c0014a0 con 0x7f5eac0b4330 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.793+0000 7f5e93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.793+0000 7f5e93fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.793+0000 7f5e93fff640 10 monclient(hunting): dump: 2024-03-29T10:49:16.776 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:16.777 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:16.777 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:16.777 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:16.777 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:16.777 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:16.777 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:16.777 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:16.777 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:16.777 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:16.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231571715 0 0) 0x7f5eac0ad7e0 con 0x7f5eac1470b0 2024-03-29T10:49:16.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:16.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:16.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:16.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 10 monclient(hunting): my global_id is 4395 2024-03-29T10:49:16.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e98004eb0 con 0x7f5eac1470b0 2024-03-29T10:49:16.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3204594896 0 0) 0x7f5eac0ac070 con 0x7f5eac1439b0 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 >> v1:172.21.15.98:6790/0 conn(0x7f5eac0b4330 legacy=0x7f5eac142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 >> v1:172.21.15.98:6789/0 conn(0x7f5eac1470b0 legacy=0x7f5eac2006d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 1 monclient: found mon.b 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac0abe40 con 0x7f5eac1439b0 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 10 monclient: _finish_auth 0 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5e93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5eb2154640 5 monclient: authenticate success, global_id 4390 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5eb2154640 10 monclient: _renew_subs 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5eb2154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:16.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5eb2154640 1 -- 172.21.15.98:0/2548582520 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eac0ac5d0 con 0x7f5eac1439b0 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5eb2154640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5eb2154640 10 monclient: _renew_subs 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5eb2154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5eb2154640 1 -- 172.21.15.98:0/2548582520 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eac143170 con 0x7f5eac1439b0 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.794+0000 7f5eb2154640 1 librados: init done 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.795+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ea4003970 con 0x7f5eac1439b0 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.795+0000 7f5e93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.795+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5ea4003c80 con 0x7f5eac1439b0 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.795+0000 7f5e93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.795+0000 7f5e93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.795+0000 7f5e93fff640 10 monclient: dump: 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:16.780 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:16.781 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:16.781 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:16.781 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:16.781 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:16.781 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:16.781 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:16.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.796+0000 7f5ea9ffb640 4 set_mon_vals no callback set 2024-03-29T10:49:16.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.797+0000 7f5eb2154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:16.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.797+0000 7f5eb2154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:16.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.797+0000 7f5eb2154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:16.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.797+0000 7f5eb2154640 1 -- 172.21.15.98:0/2548582520 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8c005080 con 0x7f5eac1439b0 2024-03-29T10:49:16.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.799+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f5ea4019c50 con 0x7f5eac1439b0 2024-03-29T10:49:16.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.799+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f5ea4083ab0 con 0x7f5eac1439b0 2024-03-29T10:49:16.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.799+0000 7f5e93fff640 10 client.4390.objecter ms_dispatch 0x7f5eac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:16.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.799+0000 7f5e93fff640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:16.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.799+0000 7f5e93fff640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:16.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.800+0000 7f5e93fff640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-29T10:49:16.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.800+0000 7f5eab7fe640 10 monclient: get_auth_request con 0x7f5e9806e020 auth_method 0 2024-03-29T10:49:16.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.800+0000 7f5e93fff640 10 client.4390.objecter ms_handle_connect 0x7f5e9806e020 2024-03-29T10:49:16.790 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.808+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ea404f390 con 0x7f5eac1439b0 2024-03-29T10:49:16.790 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.808+0000 7f5e93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:16.791 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:16.808+0000 7f5e93fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:17.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.128+0000 7f5eb2154640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:17.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.128+0000 7f5eb2154640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:17.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.128+0000 7f5eb2154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:17.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.128+0000 7f5eb2154640 1 -- 172.21.15.98:0/2548582520 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e8c005600 con 0x7f5eac1439b0 2024-03-29T10:49:17.112 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.130+0000 7f5e93fff640 1 -- 172.21.15.98:0/2548582520 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f5ea40512c0 con 0x7f5eac1439b0 2024-03-29T10:49:17.112 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.130+0000 7f5e93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:17.112 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.130+0000 7f5e93fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:17.116 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:49:17.116 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.137+0000 7f5eb2154640 10 librados: watch_flush enter 2024-03-29T10:49:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.137+0000 7f5eb2154640 10 librados: watch_flush exit 2024-03-29T10:49:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.137+0000 7f5eb2154640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-29T10:49:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.137+0000 7f5eb2154640 10 client.4390.objecter successfully canceled tick 2024-03-29T10:49:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.137+0000 7f5eb2154640 1 -- 172.21.15.98:0/2548582520 >> v1:172.21.15.98:6816/1481799426 conn(0x7f5e9806e020 legacy=0x7f5e980704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.137+0000 7f5eb2154640 10 monclient: shutdown 2024-03-29T10:49:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.137+0000 7f5eb2154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:17.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.137+0000 7f5eb2154640 1 -- 172.21.15.98:0/2548582520 >> v1:172.21.15.189:6789/0 conn(0x7f5eac1439b0 legacy=0x7f5eac0b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:17.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.137+0000 7f5eb2154640 1 -- 172.21.15.98:0/2548582520 shutdown_connections 2024-03-29T10:49:17.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.137+0000 7f5eb2154640 1 -- 172.21.15.98:0/2548582520 >> 172.21.15.98:0/2548582520 conn(0x7f5eac066a00 msgr2=0x7f5eac1d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:17.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.138+0000 7f5eb2154640 1 -- 172.21.15.98:0/2548582520 shutdown_connections 2024-03-29T10:49:17.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.138+0000 7f5eb2154640 1 -- 172.21.15.98:0/2548582520 wait complete. 2024-03-29T10:49:17.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:17.138+0000 7f5eb2154640 1 librados: shutdown 2024-03-29T10:49:17.140 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:23.946 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:24.097 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.115+0000 7fe7c7f25640 10 monclient: get_monmap_and_config 2024-03-29T10:49:24.097 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.115+0000 7fe7c7f25640 10 monclient: build_initial_monmap 2024-03-29T10:49:24.097 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.115+0000 7fe7c7f25640 10 monclient: monmap: 2024-03-29T10:49:24.097 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:24.097 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:24.097 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:49:24.115977+0000 2024-03-29T10:49:24.098 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:49:24.115977+0000 2024-03-29T10:49:24.098 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:24.098 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:24.098 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:24.098 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:49:24.098 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:49:24.098 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:24.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.115+0000 7fe7c7f25640 1 Processor -- start 2024-03-29T10:49:24.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.116+0000 7fe7c7f25640 1 -- start start 2024-03-29T10:49:24.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.116+0000 7fe7c7f25640 10 monclient: init 2024-03-29T10:49:24.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.117+0000 7fe7c7f25640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:24.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.117+0000 7fe7c7f25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:24.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.117+0000 7fe7c7f25640 10 monclient(hunting): picked mon.noname-b con 0x7fe7c00b0770 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:24.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.117+0000 7fe7c7f25640 10 monclient(hunting): picked mon.noname-a con 0x7fe7c014a700 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:24.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.117+0000 7fe7c7f25640 10 monclient(hunting): picked mon.noname-c con 0x7fe7c00ac5a0 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:24.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.117+0000 7fe7c7f25640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c00aba70 con 0x7fe7c014a700 2024-03-29T10:49:24.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.117+0000 7fe7c7f25640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c00aecb0 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.117+0000 7fe7c7f25640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c00af150 con 0x7fe7c00b0770 2024-03-29T10:49:24.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.117+0000 7fe7c7f25640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:24.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.117+0000 7fe7c7f25640 10 monclient(hunting): authenticate will time out at 1184.314332s 2024-03-29T10:49:24.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.120+0000 7fe7c5cc1640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fe7c00b0770 0x7fe7c0149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:54370/0 (socket says 172.21.15.98:54370) 2024-03-29T10:49:24.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.120+0000 7fe7c5cc1640 1 -- 172.21.15.98:0/1969882919 learned_addr learned my addr 172.21.15.98:0/1969882919 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:24.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.120+0000 7fe7c4cbf640 1 -- 172.21.15.98:0/1969882919 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169812816 0 0) 0x7fe7c00af150 con 0x7fe7c00b0770 2024-03-29T10:49:24.109 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.120+0000 7fe7c4cbf640 10 monclient(hunting): _negotiate 2024-03-29T10:49:24.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.120+0000 7fe7c4cbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:24.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.120+0000 7fe7c4cbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:24.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.120+0000 7fe7c4cbf640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:49:24.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.120+0000 7fe7c4cbf640 1 -- 172.21.15.98:0/1969882919 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7ac003520 con 0x7fe7c00b0770 2024-03-29T10:49:24.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 1 -- 172.21.15.98:0/1969882919 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714757818 0 0) 0x7fe7c00aecb0 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 10 monclient(hunting): _negotiate 2024-03-29T10:49:24.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:24.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:24.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 10 monclient(hunting): my global_id is 4393 2024-03-29T10:49:24.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 1 -- 172.21.15.98:0/1969882919 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c00af150 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 1 -- 172.21.15.98:0/1969882919 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3831175730 0 0) 0x7fe7c00af150 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 1 -- 172.21.15.98:0/1969882919 >> v1:172.21.15.98:6790/0 conn(0x7fe7c00b0770 legacy=0x7fe7c0149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:24.110 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 1 -- 172.21.15.98:0/1969882919 >> v1:172.21.15.98:6789/0 conn(0x7fe7c014a700 legacy=0x7fe7c00abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 10 monclient: _finish_hunting 0 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 1 monclient: found mon.noname-c 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 1 -- 172.21.15.98:0/1969882919 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c00ac9d0 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 10 monclient: _finish_auth 0 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 10 monclient: _check_auth_tickets 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.121+0000 7fe7c4cbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.122+0000 7fe7c4cbf640 1 -- 172.21.15.98:0/1969882919 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe7bc001630 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.122+0000 7fe7c4cbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.122+0000 7fe7c4cbf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.122+0000 7fe7c4cbf640 10 monclient: dump: 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:24.111 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.122+0000 7fe7c7f25640 5 monclient: authenticate success, global_id 4393 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.122+0000 7fe7c7f25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.122+0000 7fe7c4cbf640 1 -- 172.21.15.98:0/1969882919 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7bc001880 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.122+0000 7fe7c4cbf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.122+0000 7fe7c7f25640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.122+0000 7fe7c7f25640 4 set_mon_vals no callback set 2024-03-29T10:49:24.112 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.123+0000 7fe7c7f25640 10 monclient: shutdown 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.123+0000 7fe7c7f25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.123+0000 7fe7c7f25640 1 -- 172.21.15.98:0/1969882919 >> v1:172.21.15.189:6789/0 conn(0x7fe7c00ac5a0 legacy=0x7fe7c0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.123+0000 7fe7c4cbf640 1 -- 172.21.15.98:0/1969882919 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe7bc001c10 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.123+0000 7fe7c4cbf640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.123+0000 7fe7c7f25640 1 -- 172.21.15.98:0/1969882919 shutdown_connections 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.123+0000 7fe7c7f25640 1 -- 172.21.15.98:0/1969882919 >> 172.21.15.98:0/1969882919 conn(0x7fe7c006b660 msgr2=0x7fe7c006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.124+0000 7fe7c7f25640 1 -- 172.21.15.98:0/1969882919 shutdown_connections 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.124+0000 7fe7c7f25640 1 -- 172.21.15.98:0/1969882919 wait complete. 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.125+0000 7fe7c7f25640 10 monclient: build_initial_monmap 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.125+0000 7fe7c7f25640 10 monclient: monmap: 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:49:24.113 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.126+0000 7fe7c7f25640 1 librados: starting msgr at 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.126+0000 7fe7c7f25640 1 librados: starting objecter 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.126+0000 7fe7c7f25640 1 Processor -- start 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.126+0000 7fe7c7f25640 1 -- start start 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.126+0000 7fe7c7f25640 1 librados: setting wanted keys 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.126+0000 7fe7c7f25640 1 librados: calling monclient init 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.126+0000 7fe7c7f25640 10 monclient: init 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.127+0000 7fe7c7f25640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:24.114 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.127+0000 7fe7c7f25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:24.115 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.127+0000 7fe7c7f25640 10 monclient(hunting): picked mon.noname-c con 0x7fe7c00ac5a0 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:24.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.129+0000 7fe7c7f25640 10 monclient(hunting): picked mon.noname-b con 0x7fe7c00b0770 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:24.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.129+0000 7fe7c7f25640 10 monclient(hunting): picked mon.noname-a con 0x7fe7c014a700 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.129+0000 7fe7c7f25640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c012a450 con 0x7fe7c014a700 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.129+0000 7fe7c7f25640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c01280d0 con 0x7fe7c00b0770 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.129+0000 7fe7c7f25640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c0147b00 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.129+0000 7fe7c7f25640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.129+0000 7fe7c7f25640 10 monclient(hunting): authenticate will time out at 1184.326767s 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.130+0000 7fe7c5cc1640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fe7c00ac5a0 0x7fe7c01285f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:33290/0 (socket says 172.21.15.98:33290) 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.130+0000 7fe7c5cc1640 1 -- 172.21.15.98:0/351877240 learned_addr learned my addr 172.21.15.98:0/351877240 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.131+0000 7fe7b67fc640 10 client.?.objecter ms_handle_connect 0x7fe7c00ac5a0 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.131+0000 7fe7b67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.131+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776756251 0 0) 0x7fe7c0147b00 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.131+0000 7fe7b67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.131+0000 7fe7b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.131+0000 7fe7b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:24.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.131+0000 7fe7b67fc640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.131+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7a4004eb0 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 396463623 0 0) 0x7fe7a4004eb0 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7c0147b00 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe7b8001cd0 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 10 monclient(hunting): dump: 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:24.118 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1145556103 0 0) 0x7fe7c0147b00 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 >> v1:172.21.15.189:6789/0 conn(0x7fe7c00b0770 legacy=0x7fe7c0128d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 >> v1:172.21.15.98:6789/0 conn(0x7fe7c014a700 legacy=0x7fe7c01e5f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 1 monclient: found mon.c 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c0127d60 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 10 monclient: _finish_auth 0 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:49:24.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7c7f25640 5 monclient: authenticate success, global_id 4406 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7c7f25640 10 monclient: _renew_subs 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7c7f25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7c7f25640 1 -- 172.21.15.98:0/351877240 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c01df640 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7c7f25640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7c7f25640 10 monclient: _renew_subs 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7c7f25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7c7f25640 1 -- 172.21.15.98:0/351877240 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c01dedb0 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.132+0000 7fe7c7f25640 1 librados: init done 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.134+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7b8001ff0 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.134+0000 7fe7b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.134+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe7b8004fb0 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.134+0000 7fe7b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:24.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.134+0000 7fe7b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.134+0000 7fe7b67fc640 10 monclient: dump: 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.134+0000 7fe7c7f25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.134+0000 7fe7c7f25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.134+0000 7fe7c7f25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:24.121 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.134+0000 7fe7c7f25640 1 -- 172.21.15.98:0/351877240 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c01def40 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.134+0000 7fe7b7fff640 4 set_mon_vals no callback set 2024-03-29T10:49:24.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.135+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fe7b8005110 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.136+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fe7b8084ef0 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.136+0000 7fe7b67fc640 10 client.4406.objecter ms_dispatch 0x7fe7c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:24.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.136+0000 7fe7b67fc640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:24.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.136+0000 7fe7b67fc640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:24.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.136+0000 7fe7b67fc640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T10:49:24.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.138+0000 7fe7c54c0640 10 monclient: get_auth_request con 0x7fe7a406dd10 auth_method 0 2024-03-29T10:49:24.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.139+0000 7fe7b67fc640 10 client.4406.objecter ms_handle_connect 0x7fe7a406dd10 2024-03-29T10:49:24.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.144+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7b80507d0 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.144+0000 7fe7b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:24.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.144+0000 7fe7b67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:24.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.467+0000 7fe7c7f25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:24.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.467+0000 7fe7c7f25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:24.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.467+0000 7fe7c7f25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:24.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.467+0000 7fe7c7f25640 1 -- 172.21.15.98:0/351877240 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7c0075f10 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.471+0000 7fe7b67fc640 1 -- 172.21.15.98:0/351877240 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fe7b8052700 con 0x7fe7c00ac5a0 2024-03-29T10:49:24.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.471+0000 7fe7b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:24.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.471+0000 7fe7b67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:24.453 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:49:24.454 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:24.456 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.474+0000 7fe7c7f25640 10 librados: watch_flush enter 2024-03-29T10:49:24.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.475+0000 7fe7c7f25640 10 librados: watch_flush exit 2024-03-29T10:49:24.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.475+0000 7fe7c7f25640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T10:49:24.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.475+0000 7fe7c7f25640 10 client.4406.objecter successfully canceled tick 2024-03-29T10:49:24.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.475+0000 7fe7c7f25640 1 -- 172.21.15.98:0/351877240 >> v1:172.21.15.98:6816/1481799426 conn(0x7fe7a406dd10 legacy=0x7fe7a40701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:24.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.475+0000 7fe7c7f25640 10 monclient: shutdown 2024-03-29T10:49:24.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.475+0000 7fe7c7f25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:24.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.475+0000 7fe7c7f25640 1 -- 172.21.15.98:0/351877240 >> v1:172.21.15.98:6790/0 conn(0x7fe7c00ac5a0 legacy=0x7fe7c01285f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:24.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.475+0000 7fe7c7f25640 1 -- 172.21.15.98:0/351877240 shutdown_connections 2024-03-29T10:49:24.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.475+0000 7fe7c7f25640 1 -- 172.21.15.98:0/351877240 >> 172.21.15.98:0/351877240 conn(0x7fe7c006b660 msgr2=0x7fe7c00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:24.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.475+0000 7fe7c7f25640 1 -- 172.21.15.98:0/351877240 shutdown_connections 2024-03-29T10:49:24.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.475+0000 7fe7c7f25640 1 -- 172.21.15.98:0/351877240 wait complete. 2024-03-29T10:49:24.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:24.476+0000 7fe7c7f25640 1 librados: shutdown 2024-03-29T10:49:24.479 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:31.286 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:31.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.452+0000 7fe7511dd640 10 monclient: get_monmap_and_config 2024-03-29T10:49:31.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.452+0000 7fe7511dd640 10 monclient: build_initial_monmap 2024-03-29T10:49:31.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.452+0000 7fe7511dd640 10 monclient: monmap: 2024-03-29T10:49:31.434 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:31.434 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:31.435 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:49:31.453541+0000 2024-03-29T10:49:31.435 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:49:31.453541+0000 2024-03-29T10:49:31.435 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:31.435 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:31.435 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:31.435 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:49:31.435 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:49:31.435 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:31.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.454+0000 7fe7511dd640 1 Processor -- start 2024-03-29T10:49:31.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.454+0000 7fe7511dd640 1 -- start start 2024-03-29T10:49:31.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.454+0000 7fe7511dd640 10 monclient: init 2024-03-29T10:49:31.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.455+0000 7fe7511dd640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:31.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.455+0000 7fe7511dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:31.437 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.455+0000 7fe7511dd640 10 monclient(hunting): picked mon.noname-b con 0x7fe74c1457f0 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:31.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.456+0000 7fe74ad9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fe74c1457f0 0x7fe74c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:60120/0 (socket says 172.21.15.98:60120) 2024-03-29T10:49:31.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.456+0000 7fe7511dd640 10 monclient(hunting): picked mon.noname-a con 0x7fe74c13f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:31.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.456+0000 7fe7511dd640 10 monclient(hunting): picked mon.noname-c con 0x7fe74c0a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:31.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.456+0000 7fe7511dd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c0aaef0 con 0x7fe74c13f650 2024-03-29T10:49:31.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.456+0000 7fe7511dd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c145bd0 con 0x7fe74c0a9d70 2024-03-29T10:49:31.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.456+0000 7fe7511dd640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c145eb0 con 0x7fe74c1457f0 2024-03-29T10:49:31.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.456+0000 7fe7511dd640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:31.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.456+0000 7fe7511dd640 10 monclient(hunting): authenticate will time out at 1191.653643s 2024-03-29T10:49:31.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.457+0000 7fe74ad9d640 1 -- 172.21.15.98:0/556786921 learned_addr learned my addr 172.21.15.98:0/556786921 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:31.443 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 1 -- 172.21.15.98:0/556786921 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872256147 0 0) 0x7fe74c145eb0 con 0x7fe74c1457f0 2024-03-29T10:49:31.443 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:49:31.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:31.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:31.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:49:31.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 1 -- 172.21.15.98:0/556786921 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe734003520 con 0x7fe74c1457f0 2024-03-29T10:49:31.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 1 -- 172.21.15.98:0/556786921 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742392370 0 0) 0x7fe74c0aaef0 con 0x7fe74c13f650 2024-03-29T10:49:31.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:49:31.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:31.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:31.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 10 monclient(hunting): my global_id is 4398 2024-03-29T10:49:31.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 1 -- 172.21.15.98:0/556786921 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c145eb0 con 0x7fe74c13f650 2024-03-29T10:49:31.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 1 -- 172.21.15.98:0/556786921 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389609566 0 0) 0x7fe74c145bd0 con 0x7fe74c0a9d70 2024-03-29T10:49:31.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:49:31.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:31.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:31.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 10 monclient(hunting): my global_id is 4396 2024-03-29T10:49:31.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 1 -- 172.21.15.98:0/556786921 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c0aaef0 con 0x7fe74c0a9d70 2024-03-29T10:49:31.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.460+0000 7fe749d9b640 1 -- 172.21.15.98:0/556786921 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 658710452 0 0) 0x7fe734003520 con 0x7fe74c1457f0 2024-03-29T10:49:31.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 1 -- 172.21.15.98:0/556786921 >> v1:172.21.15.189:6789/0 conn(0x7fe74c0a9d70 legacy=0x7fe74c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:31.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 1 -- 172.21.15.98:0/556786921 >> v1:172.21.15.98:6789/0 conn(0x7fe74c13f650 legacy=0x7fe74c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:31.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:49:31.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 1 monclient: found mon.noname-b 2024-03-29T10:49:31.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:31.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:49:31.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 1 -- 172.21.15.98:0/556786921 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c146340 con 0x7fe74c1457f0 2024-03-29T10:49:31.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 10 monclient: _finish_auth 0 2024-03-29T10:49:31.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:49:31.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:31.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 1 -- 172.21.15.98:0/556786921 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe740001690 con 0x7fe74c1457f0 2024-03-29T10:49:31.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:31.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:31.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 10 monclient: dump: 2024-03-29T10:49:31.446 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:31.446 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:31.446 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:31.446 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:31.447 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:31.447 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:31.447 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:31.447 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:31.447 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:31.447 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:31.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe7511dd640 5 monclient: authenticate success, global_id 4409 2024-03-29T10:49:31.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe7511dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:31.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 1 -- 172.21.15.98:0/556786921 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7400017f0 con 0x7fe74c1457f0 2024-03-29T10:49:31.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:31.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe7511dd640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:31.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe7511dd640 4 set_mon_vals no callback set 2024-03-29T10:49:31.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 1 -- 172.21.15.98:0/556786921 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe740001c00 con 0x7fe74c1457f0 2024-03-29T10:49:31.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:31.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:31.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.461+0000 7fe749d9b640 10 monclient: dump: 2024-03-29T10:49:31.448 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:31.448 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:31.448 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:31.448 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:31.448 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:31.448 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:31.448 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:31.448 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:31.448 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:31.449 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:31.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.462+0000 7fe7511dd640 10 monclient: shutdown 2024-03-29T10:49:31.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.462+0000 7fe7511dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:31.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.462+0000 7fe7511dd640 1 -- 172.21.15.98:0/556786921 >> v1:172.21.15.98:6790/0 conn(0x7fe74c1457f0 legacy=0x7fe74c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:31.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.464+0000 7fe7511dd640 1 -- 172.21.15.98:0/556786921 shutdown_connections 2024-03-29T10:49:31.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.464+0000 7fe7511dd640 1 -- 172.21.15.98:0/556786921 >> 172.21.15.98:0/556786921 conn(0x7fe74c066a00 msgr2=0x7fe74c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:31.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.465+0000 7fe7511dd640 1 -- 172.21.15.98:0/556786921 shutdown_connections 2024-03-29T10:49:31.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.465+0000 7fe7511dd640 1 -- 172.21.15.98:0/556786921 wait complete. 2024-03-29T10:49:31.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.466+0000 7fe7511dd640 10 monclient: build_initial_monmap 2024-03-29T10:49:31.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.466+0000 7fe7511dd640 10 monclient: monmap: 2024-03-29T10:49:31.451 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:31.451 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:31.451 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:49:31.451 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:49:31.451 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:31.452 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:31.452 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:31.452 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:49:31.452 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:49:31.452 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:31.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.467+0000 7fe7511dd640 1 librados: starting msgr at 2024-03-29T10:49:31.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.467+0000 7fe7511dd640 1 librados: starting objecter 2024-03-29T10:49:31.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.467+0000 7fe7511dd640 1 Processor -- start 2024-03-29T10:49:31.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.467+0000 7fe7511dd640 1 -- start start 2024-03-29T10:49:31.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.467+0000 7fe7511dd640 1 librados: setting wanted keys 2024-03-29T10:49:31.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.467+0000 7fe7511dd640 1 librados: calling monclient init 2024-03-29T10:49:31.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.467+0000 7fe7511dd640 10 monclient: init 2024-03-29T10:49:31.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.468+0000 7fe7511dd640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:31.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.468+0000 7fe7511dd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:31.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.468+0000 7fe7511dd640 10 monclient(hunting): picked mon.noname-c con 0x7fe74c0a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:31.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.471+0000 7fe74ad9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fe74c0a9d70 0x7fe74c06e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:38390/0 (socket says 172.21.15.98:38390) 2024-03-29T10:49:31.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.472+0000 7fe7511dd640 10 monclient(hunting): picked mon.noname-b con 0x7fe74c13f650 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:31.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.472+0000 7fe74ad9d640 1 -- 172.21.15.98:0/173044653 learned_addr learned my addr 172.21.15.98:0/173044653 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:31.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.472+0000 7fe7511dd640 10 monclient(hunting): picked mon.noname-a con 0x7fe74c13b350 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:31.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.472+0000 7fe7511dd640 1 -- 172.21.15.98:0/173044653 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c13bcf0 con 0x7fe74c13b350 2024-03-29T10:49:31.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.472+0000 7fe7511dd640 1 -- 172.21.15.98:0/173044653 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c06e810 con 0x7fe74c13f650 2024-03-29T10:49:31.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.472+0000 7fe7511dd640 1 -- 172.21.15.98:0/173044653 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c1dc920 con 0x7fe74c0a9d70 2024-03-29T10:49:31.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.472+0000 7fe7511dd640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:31.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.472+0000 7fe7511dd640 10 monclient(hunting): authenticate will time out at 1191.669413s 2024-03-29T10:49:31.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.473+0000 7fe7337fe640 10 client.?.objecter ms_handle_connect 0x7fe74c0a9d70 2024-03-29T10:49:31.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.473+0000 7fe7337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:31.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.473+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486120408 0 0) 0x7fe74c1dc920 con 0x7fe74c0a9d70 2024-03-29T10:49:31.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.473+0000 7fe7337fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:31.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.474+0000 7fe7337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:31.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.474+0000 7fe7337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:31.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.474+0000 7fe7337fe640 10 monclient(hunting): my global_id is 4412 2024-03-29T10:49:31.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.474+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe738004eb0 con 0x7fe74c0a9d70 2024-03-29T10:49:31.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.474+0000 7fe7337fe640 10 client.?.objecter ms_handle_connect 0x7fe74c13f650 2024-03-29T10:49:31.458 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.474+0000 7fe7337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:31.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.474+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087586347 0 0) 0x7fe74c06e810 con 0x7fe74c13f650 2024-03-29T10:49:31.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.474+0000 7fe7337fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:31.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.474+0000 7fe7337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:31.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.474+0000 7fe7337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:31.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.474+0000 7fe7337fe640 10 monclient(hunting): my global_id is 4399 2024-03-29T10:49:31.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.475+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe74c1dc920 con 0x7fe74c13f650 2024-03-29T10:49:31.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.475+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1207193075 0 0) 0x7fe74c1dc920 con 0x7fe74c13f650 2024-03-29T10:49:31.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.475+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe74c06e810 con 0x7fe74c13f650 2024-03-29T10:49:31.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.475+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe73c0017f0 con 0x7fe74c13f650 2024-03-29T10:49:31.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.476+0000 7fe7337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:31.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.476+0000 7fe7337fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:31.459 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.476+0000 7fe7337fe640 10 monclient(hunting): dump: 2024-03-29T10:49:31.459 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:31.460 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:31.460 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:31.460 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:31.460 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:31.460 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:31.460 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:31.460 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:31.460 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:31.460 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:31.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.476+0000 7fe7337fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:31.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.476+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3964638428 0 0) 0x7fe74c06e810 con 0x7fe74c13f650 2024-03-29T10:49:31.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.476+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 >> v1:172.21.15.98:6790/0 conn(0x7fe74c0a9d70 legacy=0x7fe74c06e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:31.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.476+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 >> v1:172.21.15.98:6789/0 conn(0x7fe74c13b350 legacy=0x7fe74c137450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:31.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:31.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 1 monclient: found mon.b 2024-03-29T10:49:31.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:31.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:31.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c1dcab0 con 0x7fe74c13f650 2024-03-29T10:49:31.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:31.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:31.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:31.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7511dd640 5 monclient: authenticate success, global_id 4399 2024-03-29T10:49:31.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7511dd640 10 monclient: _renew_subs 2024-03-29T10:49:31.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7511dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:31.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7511dd640 1 -- 172.21.15.98:0/173044653 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74c138d30 con 0x7fe74c13f650 2024-03-29T10:49:31.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7511dd640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:31.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7511dd640 10 monclient: _renew_subs 2024-03-29T10:49:31.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7511dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:31.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7511dd640 1 -- 172.21.15.98:0/173044653 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74c137d40 con 0x7fe74c13f650 2024-03-29T10:49:31.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7511dd640 1 librados: init done 2024-03-29T10:49:31.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe73c001950 con 0x7fe74c13f650 2024-03-29T10:49:31.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:31.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe73c0037b0 con 0x7fe74c13f650 2024-03-29T10:49:31.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:31.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:31.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.477+0000 7fe7337fe640 10 monclient: dump: 2024-03-29T10:49:31.465 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:31.465 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:31.465 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:31.466 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:31.466 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:31.466 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:31.466 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:31.466 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:31.466 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:31.466 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:31.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.478+0000 7fe74959a640 4 set_mon_vals no callback set 2024-03-29T10:49:31.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.479+0000 7fe7511dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:31.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.479+0000 7fe7511dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:31.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.479+0000 7fe7511dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:31.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.479+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fe73c019780 con 0x7fe74c13f650 2024-03-29T10:49:31.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.480+0000 7fe7511dd640 1 -- 172.21.15.98:0/173044653 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72c005080 con 0x7fe74c13f650 2024-03-29T10:49:31.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.481+0000 7fe74ad9d640 10 monclient: get_auth_request con 0x7fe73806ddc0 auth_method 0 2024-03-29T10:49:31.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.481+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fe73c083750 con 0x7fe74c13f650 2024-03-29T10:49:31.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.481+0000 7fe7337fe640 10 client.4399.objecter ms_dispatch 0x7fe74c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:31.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.481+0000 7fe7337fe640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:31.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.481+0000 7fe7337fe640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:31.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.481+0000 7fe7337fe640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-29T10:49:31.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.482+0000 7fe7337fe640 10 client.4399.objecter ms_handle_connect 0x7fe73806ddc0 2024-03-29T10:49:31.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.488+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe73c04f030 con 0x7fe74c13f650 2024-03-29T10:49:31.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.488+0000 7fe7337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:31.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.488+0000 7fe7337fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:31.794 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.812+0000 7fe7511dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:31.794 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.812+0000 7fe7511dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:31.794 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.812+0000 7fe7511dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:31.794 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.812+0000 7fe7511dd640 1 -- 172.21.15.98:0/173044653 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe72c005330 con 0x7fe74c13f650 2024-03-29T10:49:31.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.813+0000 7fe7337fe640 1 -- 172.21.15.98:0/173044653 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fe73c050f60 con 0x7fe74c13f650 2024-03-29T10:49:31.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.813+0000 7fe7337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:31.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.813+0000 7fe7337fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:31.795 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:49:31.795 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:31.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.816+0000 7fe7511dd640 10 librados: watch_flush enter 2024-03-29T10:49:31.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.817+0000 7fe7511dd640 10 librados: watch_flush exit 2024-03-29T10:49:31.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.817+0000 7fe7511dd640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-29T10:49:31.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.817+0000 7fe7511dd640 10 client.4399.objecter successfully canceled tick 2024-03-29T10:49:31.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.817+0000 7fe7511dd640 1 -- 172.21.15.98:0/173044653 >> v1:172.21.15.98:6816/1481799426 conn(0x7fe73806ddc0 legacy=0x7fe738070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:31.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.817+0000 7fe7511dd640 10 monclient: shutdown 2024-03-29T10:49:31.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.817+0000 7fe7511dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:31.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.817+0000 7fe7511dd640 1 -- 172.21.15.98:0/173044653 >> v1:172.21.15.189:6789/0 conn(0x7fe74c13f650 legacy=0x7fe74c1dc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:31.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.818+0000 7fe7511dd640 1 -- 172.21.15.98:0/173044653 shutdown_connections 2024-03-29T10:49:31.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.818+0000 7fe7511dd640 1 -- 172.21.15.98:0/173044653 >> 172.21.15.98:0/173044653 conn(0x7fe74c066a00 msgr2=0x7fe74c0b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:31.800 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.818+0000 7fe7511dd640 1 -- 172.21.15.98:0/173044653 shutdown_connections 2024-03-29T10:49:31.800 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.818+0000 7fe7511dd640 1 -- 172.21.15.98:0/173044653 wait complete. 2024-03-29T10:49:31.800 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:31.819+0000 7fe7511dd640 1 librados: shutdown 2024-03-29T10:49:31.817 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:38.625 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:38.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.790+0000 7f8af5f9d640 10 monclient: get_monmap_and_config 2024-03-29T10:49:38.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.790+0000 7f8af5f9d640 10 monclient: build_initial_monmap 2024-03-29T10:49:38.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.790+0000 7f8af5f9d640 10 monclient: monmap: 2024-03-29T10:49:38.772 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:38.772 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:38.772 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:49:38.791429+0000 2024-03-29T10:49:38.772 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:49:38.791429+0000 2024-03-29T10:49:38.773 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:38.773 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:38.773 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:38.773 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:49:38.773 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:49:38.773 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:38.774 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.793+0000 7f8af5f9d640 1 Processor -- start 2024-03-29T10:49:38.774 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.793+0000 7f8af5f9d640 1 -- start start 2024-03-29T10:49:38.775 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.793+0000 7f8af5f9d640 10 monclient: init 2024-03-29T10:49:38.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.793+0000 7f8af5f9d640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:38.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.793+0000 7f8af5f9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:38.777 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.794+0000 7f8af5f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8af00b4330 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:38.777 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.794+0000 7f8af5f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8af01470b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:38.777 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.794+0000 7f8af5f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f8af01439b0 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:38.777 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.794+0000 7f8af5f9d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af006f2a0 con 0x7f8af01439b0 2024-03-29T10:49:38.777 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.794+0000 7f8af5f9d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af0146b90 con 0x7f8af01470b0 2024-03-29T10:49:38.777 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.794+0000 7f8af5f9d640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af00b2ca0 con 0x7f8af00b4330 2024-03-29T10:49:38.777 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.794+0000 7f8af5f9d640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:38.777 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.794+0000 7f8af5f9d640 10 monclient(hunting): authenticate will time out at 1198.991040s 2024-03-29T10:49:38.777 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.794+0000 7f8aef7fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f8af00b4330 0x7f8af00b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:58194/0 (socket says 172.21.15.98:58194) 2024-03-29T10:49:38.777 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.794+0000 7f8aef7fe640 1 -- 172.21.15.98:0/2563734522 learned_addr learned my addr 172.21.15.98:0/2563734522 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:38.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.795+0000 7f8aee7fc640 1 -- 172.21.15.98:0/2563734522 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807122589 0 0) 0x7f8af00b2ca0 con 0x7f8af00b4330 2024-03-29T10:49:38.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.795+0000 7f8aee7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:38.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.795+0000 7f8aee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:38.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.795+0000 7f8aee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:38.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.795+0000 7f8aee7fc640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:49:38.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.795+0000 7f8aee7fc640 1 -- 172.21.15.98:0/2563734522 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad8003520 con 0x7f8af00b4330 2024-03-29T10:49:38.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.796+0000 7f8aee7fc640 1 -- 172.21.15.98:0/2563734522 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1103219326 0 0) 0x7f8ad8003520 con 0x7f8af00b4330 2024-03-29T10:49:38.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.796+0000 7f8aee7fc640 1 -- 172.21.15.98:0/2563734522 >> v1:172.21.15.189:6789/0 conn(0x7f8af01470b0 legacy=0x7f8af00b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:38.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.797+0000 7f8aee7fc640 1 -- 172.21.15.98:0/2563734522 >> v1:172.21.15.98:6789/0 conn(0x7f8af01439b0 legacy=0x7f8af0151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8aee7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8aee7fc640 1 monclient: found mon.noname-b 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8aee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8aee7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8aee7fc640 1 -- 172.21.15.98:0/2563734522 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af006e0c0 con 0x7f8af00b4330 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8aee7fc640 10 monclient: _finish_auth 0 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8aee7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8aee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8aee7fc640 1 -- 172.21.15.98:0/2563734522 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8ae4001690 con 0x7f8af00b4330 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8aee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8aee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8aee7fc640 10 monclient: dump: 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:38.780 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:38.781 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:38.781 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:38.781 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:38.781 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:38.781 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:38.781 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:38.781 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:38.781 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:38.781 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8af5f9d640 5 monclient: authenticate success, global_id 4415 2024-03-29T10:49:38.781 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.798+0000 7f8af5f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:38.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.799+0000 7f8aee7fc640 1 -- 172.21.15.98:0/2563734522 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ae4001860 con 0x7f8af00b4330 2024-03-29T10:49:38.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.799+0000 7f8aee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:38.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.799+0000 7f8af5f9d640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:38.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.799+0000 7f8af5f9d640 4 set_mon_vals no callback set 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.800+0000 7f8aee7fc640 1 -- 172.21.15.98:0/2563734522 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8ae4001cb0 con 0x7f8af00b4330 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.800+0000 7f8aee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.800+0000 7f8aee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.800+0000 7f8aee7fc640 10 monclient: dump: 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:38.783 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:38.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.800+0000 7f8af5f9d640 10 monclient: shutdown 2024-03-29T10:49:38.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.800+0000 7f8af5f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:38.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.800+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2563734522 >> v1:172.21.15.98:6790/0 conn(0x7f8af00b4330 legacy=0x7f8af00b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:38.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.801+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2563734522 shutdown_connections 2024-03-29T10:49:38.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.801+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2563734522 >> 172.21.15.98:0/2563734522 conn(0x7f8af0066a00 msgr2=0x7f8af0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:38.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.802+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2563734522 shutdown_connections 2024-03-29T10:49:38.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.802+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2563734522 wait complete. 2024-03-29T10:49:38.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.803+0000 7f8af5f9d640 10 monclient: build_initial_monmap 2024-03-29T10:49:38.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.803+0000 7f8af5f9d640 10 monclient: monmap: 2024-03-29T10:49:38.785 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:38.785 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:38.786 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:49:38.786 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:49:38.786 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:38.786 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:38.786 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:38.786 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:49:38.786 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:49:38.786 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:38.786 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.803+0000 7f8af5f9d640 1 librados: starting msgr at 2024-03-29T10:49:38.786 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.803+0000 7f8af5f9d640 1 librados: starting objecter 2024-03-29T10:49:38.786 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.804+0000 7f8af5f9d640 1 Processor -- start 2024-03-29T10:49:38.788 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.804+0000 7f8af5f9d640 1 -- start start 2024-03-29T10:49:38.788 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.804+0000 7f8af5f9d640 1 librados: setting wanted keys 2024-03-29T10:49:38.788 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.804+0000 7f8af5f9d640 1 librados: calling monclient init 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.804+0000 7f8af5f9d640 10 monclient: init 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.805+0000 7f8af5f9d640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.805+0000 7f8af5f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.805+0000 7f8af5f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8af00b4330 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.805+0000 7f8af5f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f8af01439b0 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.805+0000 7f8af5f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8af01470b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.805+0000 7f8af5f9d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af011f910 con 0x7f8af01439b0 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.805+0000 7f8af5f9d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af01465e0 con 0x7f8af01470b0 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.805+0000 7f8af5f9d640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af01204b0 con 0x7f8af00b4330 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.805+0000 7f8af5f9d640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.805+0000 7f8af5f9d640 10 monclient(hunting): authenticate will time out at 1199.002448s 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.806+0000 7f8aef7fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f8af00b4330 0x7f8af01dee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:36994/0 (socket says 172.21.15.98:36994) 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.807+0000 7f8aef7fe640 1 -- 172.21.15.98:0/2162792753 learned_addr learned my addr 172.21.15.98:0/2162792753 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.807+0000 7f8ad7fff640 10 client.?.objecter ms_handle_connect 0x7f8af00b4330 2024-03-29T10:49:38.789 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.807+0000 7f8ad7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.807+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775066407 0 0) 0x7f8af01204b0 con 0x7f8af00b4330 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.807+0000 7f8ad7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.807+0000 7f8ad7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.807+0000 7f8ad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.807+0000 7f8ad7fff640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.807+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8adc004eb0 con 0x7f8af00b4330 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.808+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2200216966 0 0) 0x7f8adc004eb0 con 0x7f8af00b4330 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.808+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8af01204b0 con 0x7f8af00b4330 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.808+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8ae4001c30 con 0x7f8af00b4330 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.808+0000 7f8ad7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.808+0000 7f8ad7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.808+0000 7f8ad7fff640 10 monclient(hunting): dump: 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:38.792 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.808+0000 7f8ad7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.809+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2437045866 0 0) 0x7f8af01204b0 con 0x7f8af00b4330 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.809+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 >> v1:172.21.15.189:6789/0 conn(0x7f8af01470b0 legacy=0x7f8af011fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 >> v1:172.21.15.98:6789/0 conn(0x7f8af01439b0 legacy=0x7f8af01df540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 1 monclient: found mon.c 2024-03-29T10:49:38.793 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:38.794 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:38.794 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af0120640 con 0x7f8af00b4330 2024-03-29T10:49:38.794 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 10 monclient: _finish_auth 0 2024-03-29T10:49:38.794 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:38.794 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:38.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8af5f9d640 5 monclient: authenticate success, global_id 4418 2024-03-29T10:49:38.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8af5f9d640 10 monclient: _renew_subs 2024-03-29T10:49:38.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8af5f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2162792753 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af0120890 con 0x7f8af00b4330 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8af5f9d640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8af5f9d640 10 monclient: _renew_subs 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8af5f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2162792753 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af01e7360 con 0x7f8af00b4330 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8af5f9d640 1 librados: init done 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ae4004a90 con 0x7f8af00b4330 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8ae4004ee0 con 0x7f8af00b4330 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.810+0000 7f8ad7fff640 10 monclient: dump: 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:38.796 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.811+0000 7f8aedffb640 4 set_mon_vals no callback set 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.812+0000 7f8af5f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.812+0000 7f8af5f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.812+0000 7f8af5f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.812+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2162792753 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad0005080 con 0x7f8af00b4330 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.812+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f8ae4005040 con 0x7f8af00b4330 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.813+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f8ae4084bb0 con 0x7f8af00b4330 2024-03-29T10:49:38.797 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.813+0000 7f8ad7fff640 10 client.4418.objecter ms_dispatch 0x7f8af0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:38.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.813+0000 7f8ad7fff640 3 client.4418.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:38.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.813+0000 7f8aeeffd640 10 monclient: get_auth_request con 0x7f8adc06db90 auth_method 0 2024-03-29T10:49:38.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.814+0000 7f8ad7fff640 3 client.4418.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:38.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.814+0000 7f8ad7fff640 20 client.4418.objecter dump_active .. 0 homeless 2024-03-29T10:49:38.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.814+0000 7f8ad7fff640 10 client.4418.objecter ms_handle_connect 0x7f8adc06db90 2024-03-29T10:49:38.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.823+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ae40504f0 con 0x7f8af00b4330 2024-03-29T10:49:38.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.823+0000 7f8ad7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:38.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:38.823+0000 7f8ad7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:39.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.147+0000 7f8af5f9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:39.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.147+0000 7f8af5f9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:39.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.147+0000 7f8af5f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:39.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.147+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2162792753 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ad0005330 con 0x7f8af00b4330 2024-03-29T10:49:39.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.147+0000 7f8ad7fff640 1 -- 172.21.15.98:0/2162792753 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f8ae4084260 con 0x7f8af00b4330 2024-03-29T10:49:39.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.147+0000 7f8ad7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:39.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.147+0000 7f8ad7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:39.132 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:49:39.133 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:39.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.151+0000 7f8af5f9d640 10 librados: watch_flush enter 2024-03-29T10:49:39.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.151+0000 7f8af5f9d640 10 librados: watch_flush exit 2024-03-29T10:49:39.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.151+0000 7f8af5f9d640 20 client.4418.objecter shutdown clearing up homeless session... 2024-03-29T10:49:39.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.151+0000 7f8af5f9d640 10 client.4418.objecter successfully canceled tick 2024-03-29T10:49:39.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.151+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2162792753 >> v1:172.21.15.98:6816/1481799426 conn(0x7f8adc06db90 legacy=0x7f8adc070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:39.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.151+0000 7f8af5f9d640 10 monclient: shutdown 2024-03-29T10:49:39.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.151+0000 7f8af5f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:39.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.151+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2162792753 >> v1:172.21.15.98:6790/0 conn(0x7f8af00b4330 legacy=0x7f8af01dee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:39.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.151+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2162792753 shutdown_connections 2024-03-29T10:49:39.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.151+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2162792753 >> 172.21.15.98:0/2162792753 conn(0x7f8af0066a00 msgr2=0x7f8af00b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:39.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.152+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2162792753 shutdown_connections 2024-03-29T10:49:39.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.153+0000 7f8af5f9d640 1 -- 172.21.15.98:0/2162792753 wait complete. 2024-03-29T10:49:39.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:39.154+0000 7f8af5f9d640 1 librados: shutdown 2024-03-29T10:49:39.153 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:45.960 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:46.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.133+0000 7fbb5e2ac640 10 monclient: get_monmap_and_config 2024-03-29T10:49:46.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.133+0000 7fbb5e2ac640 10 monclient: build_initial_monmap 2024-03-29T10:49:46.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.133+0000 7fbb5e2ac640 10 monclient: monmap: 2024-03-29T10:49:46.116 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:46.116 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:46.116 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:49:46.133945+0000 2024-03-29T10:49:46.116 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:49:46.133945+0000 2024-03-29T10:49:46.116 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:46.117 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:46.117 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:46.117 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:49:46.117 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:49:46.117 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:46.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.134+0000 7fbb5e2ac640 1 Processor -- start 2024-03-29T10:49:46.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.134+0000 7fbb5e2ac640 1 -- start start 2024-03-29T10:49:46.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.134+0000 7fbb5e2ac640 10 monclient: init 2024-03-29T10:49:46.117 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.135+0000 7fbb5e2ac640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:46.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.135+0000 7fbb5e2ac640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:46.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.135+0000 7fbb5e2ac640 10 monclient(hunting): picked mon.noname-a con 0x7fbb50034850 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:46.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.135+0000 7fbb5e2ac640 10 monclient(hunting): picked mon.noname-c con 0x7fbb500c4ec0 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:46.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.135+0000 7fbb5e2ac640 10 monclient(hunting): picked mon.noname-b con 0x7fbb500c7d30 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:46.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.135+0000 7fbb5e2ac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb500c83c0 con 0x7fbb50034850 2024-03-29T10:49:46.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.135+0000 7fbb5e2ac640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb50033280 con 0x7fbb500c4ec0 2024-03-29T10:49:46.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.135+0000 7fbb5e2ac640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb50029100 con 0x7fbb500c7d30 2024-03-29T10:49:46.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.135+0000 7fbb5e2ac640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:46.118 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.135+0000 7fbb5e2ac640 10 monclient(hunting): authenticate will time out at 1206.332788s 2024-03-29T10:49:46.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.139+0000 7fbb5daab640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fbb500c7d30 0x7fbb50029f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:50753/0 (socket says 172.21.15.98:50753) 2024-03-29T10:49:46.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.139+0000 7fbb5daab640 1 -- 172.21.15.98:0/2440839160 learned_addr learned my addr 172.21.15.98:0/2440839160 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:46.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.139+0000 7fbb4ffff640 1 -- 172.21.15.98:0/2440839160 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021558582 0 0) 0x7fbb50029100 con 0x7fbb500c7d30 2024-03-29T10:49:46.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.139+0000 7fbb4ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:46.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.139+0000 7fbb4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:46.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.139+0000 7fbb4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:46.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.139+0000 7fbb4ffff640 10 monclient(hunting): my global_id is 4421 2024-03-29T10:49:46.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.139+0000 7fbb4ffff640 1 -- 172.21.15.98:0/2440839160 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb44003520 con 0x7fbb500c7d30 2024-03-29T10:49:46.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.139+0000 7fbb4ffff640 1 -- 172.21.15.98:0/2440839160 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3494439753 0 0) 0x7fbb44003520 con 0x7fbb500c7d30 2024-03-29T10:49:46.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.139+0000 7fbb4ffff640 1 -- 172.21.15.98:0/2440839160 >> v1:172.21.15.189:6789/0 conn(0x7fbb500c4ec0 legacy=0x7fbb50029800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:46.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 1 -- 172.21.15.98:0/2440839160 >> v1:172.21.15.98:6789/0 conn(0x7fbb50034850 legacy=0x7fbb500c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:46.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:46.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 1 monclient: found mon.noname-b 2024-03-29T10:49:46.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:46.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:49:46.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 1 -- 172.21.15.98:0/2440839160 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb500293e0 con 0x7fbb500c7d30 2024-03-29T10:49:46.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 10 monclient: _finish_auth 0 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 1 -- 172.21.15.98:0/2440839160 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fbb54001670 con 0x7fbb500c7d30 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 10 monclient: dump: 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:46.127 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb5e2ac640 5 monclient: authenticate success, global_id 4421 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb5e2ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 1 -- 172.21.15.98:0/2440839160 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb540018d0 con 0x7fbb500c7d30 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb5e2ac640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb5e2ac640 4 set_mon_vals no callback set 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 1 -- 172.21.15.98:0/2440839160 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fbb54001ca0 con 0x7fbb500c7d30 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.142+0000 7fbb4ffff640 10 monclient: dump: 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:46.128 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:46.129 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:46.129 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:46.129 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:46.129 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:46.129 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:46.129 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:46.129 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:46.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.143+0000 7fbb5e2ac640 10 monclient: shutdown 2024-03-29T10:49:46.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.143+0000 7fbb5e2ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:46.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.143+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2440839160 >> v1:172.21.15.98:6790/0 conn(0x7fbb500c7d30 legacy=0x7fbb50029f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:46.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.147+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2440839160 shutdown_connections 2024-03-29T10:49:46.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.147+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2440839160 >> 172.21.15.98:0/2440839160 conn(0x7fbb500058f0 msgr2=0x7fbb50005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:46.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.147+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2440839160 shutdown_connections 2024-03-29T10:49:46.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.147+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2440839160 wait complete. 2024-03-29T10:49:46.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.148+0000 7fbb5e2ac640 10 monclient: build_initial_monmap 2024-03-29T10:49:46.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.148+0000 7fbb5e2ac640 10 monclient: monmap: 2024-03-29T10:49:46.131 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:46.131 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:46.131 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:49:46.131 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:49:46.131 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:46.132 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:46.132 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:46.132 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:49:46.132 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:49:46.132 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:46.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.149+0000 7fbb5e2ac640 1 librados: starting msgr at 2024-03-29T10:49:46.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.149+0000 7fbb5e2ac640 1 librados: starting objecter 2024-03-29T10:49:46.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.149+0000 7fbb5e2ac640 1 Processor -- start 2024-03-29T10:49:46.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.149+0000 7fbb5e2ac640 1 -- start start 2024-03-29T10:49:46.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.149+0000 7fbb5e2ac640 1 librados: setting wanted keys 2024-03-29T10:49:46.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.149+0000 7fbb5e2ac640 1 librados: calling monclient init 2024-03-29T10:49:46.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.149+0000 7fbb5e2ac640 10 monclient: init 2024-03-29T10:49:46.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.150+0000 7fbb5e2ac640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:46.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.150+0000 7fbb5e2ac640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:46.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.153+0000 7fbb5e2ac640 10 monclient(hunting): picked mon.noname-b con 0x7fbb50034850 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:46.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.154+0000 7fbb5d2aa640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbb50034850 0x7fbb500ca480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:38045/0 (socket says 172.21.15.98:38045) 2024-03-29T10:49:46.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.154+0000 7fbb5e2ac640 10 monclient(hunting): picked mon.noname-c con 0x7fbb500c4ec0 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:46.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.154+0000 7fbb5e2ac640 10 monclient(hunting): picked mon.noname-a con 0x7fbb500c7d30 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:46.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.154+0000 7fbb5e2ac640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb500cb7d0 con 0x7fbb500c7d30 2024-03-29T10:49:46.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.154+0000 7fbb5e2ac640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb500cbab0 con 0x7fbb50034850 2024-03-29T10:49:46.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.154+0000 7fbb5e2ac640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb500cab90 con 0x7fbb500c4ec0 2024-03-29T10:49:46.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.154+0000 7fbb5e2ac640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:46.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.154+0000 7fbb5e2ac640 10 monclient(hunting): authenticate will time out at 1206.351848s 2024-03-29T10:49:46.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.155+0000 7fbb5d2aa640 1 -- 172.21.15.98:0/2973024310 learned_addr learned my addr 172.21.15.98:0/2973024310 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:46.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.155+0000 7fbb4dffb640 10 client.?.objecter ms_handle_connect 0x7fbb50034850 2024-03-29T10:49:46.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.155+0000 7fbb4dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:46.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.155+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183920614 0 0) 0x7fbb500cbab0 con 0x7fbb50034850 2024-03-29T10:49:46.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.156+0000 7fbb4dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:46.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.156+0000 7fbb4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:46.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.156+0000 7fbb4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:46.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.156+0000 7fbb4dffb640 10 monclient(hunting): my global_id is 4402 2024-03-29T10:49:46.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.156+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb3c004eb0 con 0x7fbb50034850 2024-03-29T10:49:46.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.156+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2408204002 0 0) 0x7fbb3c004eb0 con 0x7fbb50034850 2024-03-29T10:49:46.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.156+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb500cbab0 con 0x7fbb50034850 2024-03-29T10:49:46.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.156+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fbb48001890 con 0x7fbb50034850 2024-03-29T10:49:46.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.156+0000 7fbb4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:46.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.156+0000 7fbb4dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:46.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.156+0000 7fbb4dffb640 10 monclient(hunting): dump: 2024-03-29T10:49:46.139 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:46.139 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:46.139 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:46.139 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:46.139 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:46.139 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:46.139 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:46.139 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:46.139 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:46.139 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:46.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.156+0000 7fbb4dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:46.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.157+0000 7fbb4dffb640 10 client.?.objecter ms_handle_connect 0x7fbb500c4ec0 2024-03-29T10:49:46.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.157+0000 7fbb4dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:46.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.157+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68947004 0 0) 0x7fbb500cab90 con 0x7fbb500c4ec0 2024-03-29T10:49:46.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.157+0000 7fbb4dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.158+0000 7fbb4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.158+0000 7fbb4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.158+0000 7fbb4dffb640 10 monclient(hunting): my global_id is 4424 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.158+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb3c004eb0 con 0x7fbb500c4ec0 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.158+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3929981753 0 0) 0x7fbb3c004eb0 con 0x7fbb500c4ec0 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.158+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb500cab90 con 0x7fbb500c4ec0 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.158+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fbb5804fdd0 con 0x7fbb500c4ec0 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.158+0000 7fbb4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.159+0000 7fbb4dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.159+0000 7fbb4dffb640 10 monclient(hunting): dump: 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:46.142 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:46.143 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:46.143 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:46.143 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:46.143 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:46.143 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:46.143 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:46.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.159+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2581555922 0 0) 0x7fbb500cab90 con 0x7fbb500c4ec0 2024-03-29T10:49:46.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.159+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 >> v1:172.21.15.189:6789/0 conn(0x7fbb50034850 legacy=0x7fbb500ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:46.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.159+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 >> v1:172.21.15.98:6789/0 conn(0x7fbb500c7d30 legacy=0x7fbb500b2d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:46.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb4dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:49:46.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb4dffb640 1 monclient: found mon.c 2024-03-29T10:49:46.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb4dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb500bf1e0 con 0x7fbb500c4ec0 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb4dffb640 10 monclient: _finish_auth 0 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb4dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb4dffb640 10 client.?.objecter ms_handle_connect 0x7fbb500c7d30 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb4dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb5e2ac640 5 monclient: authenticate success, global_id 4424 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb5e2ac640 10 monclient: _renew_subs 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb5e2ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2973024310 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb500c9730 con 0x7fbb500c4ec0 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb5e2ac640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb5e2ac640 10 monclient: _renew_subs 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb5e2ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:46.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2973024310 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5015d4b0 con 0x7fbb500c4ec0 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.161+0000 7fbb5e2ac640 1 librados: init done 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.163+0000 7fbb5e2ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.163+0000 7fbb5e2ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.163+0000 7fbb5e2ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.163+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb5804f400 con 0x7fbb500c4ec0 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.163+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2973024310 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5015c7a0 con 0x7fbb500c4ec0 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.163+0000 7fbb4dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.163+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fbb58075710 con 0x7fbb500c4ec0 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.163+0000 7fbb4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.163+0000 7fbb4f7fe640 4 set_mon_vals no callback set 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.163+0000 7fbb4dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.163+0000 7fbb4dffb640 10 monclient: dump: 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:46.148 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:46.149 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:46.149 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:46.149 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:46.149 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:46.149 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:46.149 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:46.149 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:46.149 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:46.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.168+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fbb58067bf0 con 0x7fbb500c4ec0 2024-03-29T10:49:46.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.168+0000 7fbb5d2aa640 10 monclient: get_auth_request con 0x7fbb3c06dd20 auth_method 0 2024-03-29T10:49:46.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.169+0000 7fbb4dffb640 10 client.4424.objecter ms_handle_connect 0x7fbb3c06dd20 2024-03-29T10:49:46.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.169+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fbb580fe210 con 0x7fbb500c4ec0 2024-03-29T10:49:46.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.169+0000 7fbb4dffb640 10 client.4424.objecter ms_dispatch 0x7fbb58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:46.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.169+0000 7fbb4dffb640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:46.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.169+0000 7fbb4dffb640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:46.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.169+0000 7fbb4dffb640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T10:49:46.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.174+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb580c9af0 con 0x7fbb500c4ec0 2024-03-29T10:49:46.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.174+0000 7fbb4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:46.156 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.174+0000 7fbb4dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:46.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.504+0000 7fbb5e2ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:46.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.504+0000 7fbb5e2ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:46.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.504+0000 7fbb5e2ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:49:46.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.504+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2973024310 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb50004c50 con 0x7fbb500c4ec0 2024-03-29T10:49:46.488 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.505+0000 7fbb4dffb640 1 -- 172.21.15.98:0/2973024310 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fbb580cba20 con 0x7fbb500c4ec0 2024-03-29T10:49:46.488 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.507+0000 7fbb4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:46.488 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.507+0000 7fbb4dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:46.490 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:49:46.490 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:46.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.510+0000 7fbb5e2ac640 10 librados: watch_flush enter 2024-03-29T10:49:46.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.510+0000 7fbb5e2ac640 10 librados: watch_flush exit 2024-03-29T10:49:46.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.510+0000 7fbb5e2ac640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T10:49:46.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.510+0000 7fbb5e2ac640 10 client.4424.objecter successfully canceled tick 2024-03-29T10:49:46.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.510+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2973024310 >> v1:172.21.15.98:6816/1481799426 conn(0x7fbb3c06dd20 legacy=0x7fbb3c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:46.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.510+0000 7fbb5e2ac640 10 monclient: shutdown 2024-03-29T10:49:46.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.510+0000 7fbb5e2ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:46.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.510+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2973024310 >> v1:172.21.15.98:6790/0 conn(0x7fbb500c4ec0 legacy=0x7fbb5015cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:46.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.511+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2973024310 shutdown_connections 2024-03-29T10:49:46.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.511+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2973024310 >> 172.21.15.98:0/2973024310 conn(0x7fbb500058f0 msgr2=0x7fbb5002a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:46.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.511+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2973024310 shutdown_connections 2024-03-29T10:49:46.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.511+0000 7fbb5e2ac640 1 -- 172.21.15.98:0/2973024310 wait complete. 2024-03-29T10:49:46.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:46.511+0000 7fbb5e2ac640 1 librados: shutdown 2024-03-29T10:49:46.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:53.317 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:53.474 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.493+0000 7fe8d83b5640 10 monclient: get_monmap_and_config 2024-03-29T10:49:53.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.493+0000 7fe8d83b5640 10 monclient: build_initial_monmap 2024-03-29T10:49:53.475 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.493+0000 7fe8d83b5640 10 monclient: monmap: 2024-03-29T10:49:53.475 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:53.475 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:53.475 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:49:53.493946+0000 2024-03-29T10:49:53.475 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:49:53.493946+0000 2024-03-29T10:49:53.475 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:53.475 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:53.475 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:53.475 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:49:53.475 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:49:53.475 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:53.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.494+0000 7fe8d83b5640 1 Processor -- start 2024-03-29T10:49:53.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.494+0000 7fe8d83b5640 1 -- start start 2024-03-29T10:49:53.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.494+0000 7fe8d83b5640 10 monclient: init 2024-03-29T10:49:53.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.494+0000 7fe8d83b5640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:53.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.495+0000 7fe8d83b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:49:53.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.496+0000 7fe8d83b5640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d006ab70 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:53.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.496+0000 7fe8d83b5640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d006af50 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:53.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.496+0000 7fe8d83b5640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d0141fb0 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:53.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.496+0000 7fe8d83b5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00b0d50 con 0x7fe8d006ab70 2024-03-29T10:49:53.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.496+0000 7fe8d83b5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00b1030 con 0x7fe8d0141fb0 2024-03-29T10:49:53.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.496+0000 7fe8d83b5640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00afa70 con 0x7fe8d006af50 2024-03-29T10:49:53.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.496+0000 7fe8d83b5640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:53.478 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.496+0000 7fe8d83b5640 10 monclient(hunting): authenticate will time out at 1213.693452s 2024-03-29T10:49:53.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.500+0000 7fe8d5950640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fe8d006af50 0x7fe8d0072290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:56049/0 (socket says 172.21.15.98:56049) 2024-03-29T10:49:53.482 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.500+0000 7fe8d5950640 1 -- 172.21.15.98:0/1916381066 learned_addr learned my addr 172.21.15.98:0/1916381066 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:53.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.501+0000 7fe8d514f640 1 -- 172.21.15.98:0/1916381066 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164000284 0 0) 0x7fe8d00afa70 con 0x7fe8d006af50 2024-03-29T10:49:53.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.501+0000 7fe8d514f640 10 monclient(hunting): _negotiate 2024-03-29T10:49:53.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.501+0000 7fe8d514f640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:53.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.501+0000 7fe8d514f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:53.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.501+0000 7fe8d514f640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:49:53.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.501+0000 7fe8d514f640 1 -- 172.21.15.98:0/1916381066 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8bc003520 con 0x7fe8d006af50 2024-03-29T10:49:53.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.501+0000 7fe8d514f640 1 -- 172.21.15.98:0/1916381066 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 956603864 0 0) 0x7fe8bc003520 con 0x7fe8d006af50 2024-03-29T10:49:53.483 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.501+0000 7fe8d514f640 1 -- 172.21.15.98:0/1916381066 >> v1:172.21.15.189:6789/0 conn(0x7fe8d0141fb0 legacy=0x7fe8d00715b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:53.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d514f640 1 -- 172.21.15.98:0/1916381066 >> v1:172.21.15.98:6789/0 conn(0x7fe8d006ab70 legacy=0x7fe8d00af2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:53.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d514f640 10 monclient: _finish_hunting 0 2024-03-29T10:49:53.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d514f640 1 monclient: found mon.noname-b 2024-03-29T10:49:53.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d514f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:53.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d514f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:49:53.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d514f640 1 -- 172.21.15.98:0/1916381066 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d01423e0 con 0x7fe8d006af50 2024-03-29T10:49:53.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d514f640 10 monclient: _finish_auth 0 2024-03-29T10:49:53.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d514f640 10 monclient: _check_auth_tickets 2024-03-29T10:49:53.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d514f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:53.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d83b5640 5 monclient: authenticate success, global_id 4427 2024-03-29T10:49:53.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d83b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:53.486 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d514f640 1 -- 172.21.15.98:0/1916381066 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe8c0001690 con 0x7fe8d006af50 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d514f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d514f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d514f640 10 monclient: dump: 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:53.487 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.503+0000 7fe8d83b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:53.488 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.506+0000 7fe8d514f640 1 -- 172.21.15.98:0/1916381066 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8c0001920 con 0x7fe8d006af50 2024-03-29T10:49:53.488 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.506+0000 7fe8d514f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:53.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.506+0000 7fe8d83b5640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:53.489 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.506+0000 7fe8d83b5640 4 set_mon_vals no callback set 2024-03-29T10:49:53.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.507+0000 7fe8d514f640 1 -- 172.21.15.98:0/1916381066 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe8c0001cb0 con 0x7fe8d006af50 2024-03-29T10:49:53.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.507+0000 7fe8d514f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:53.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.507+0000 7fe8d514f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:53.490 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.507+0000 7fe8d514f640 10 monclient: dump: 2024-03-29T10:49:53.490 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:53.490 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:53.490 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:53.490 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:53.490 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:53.490 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:53.490 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:53.490 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:53.490 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:53.491 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:53.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.507+0000 7fe8d83b5640 10 monclient: shutdown 2024-03-29T10:49:53.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.507+0000 7fe8d83b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:53.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.507+0000 7fe8d83b5640 1 -- 172.21.15.98:0/1916381066 >> v1:172.21.15.98:6790/0 conn(0x7fe8d006af50 legacy=0x7fe8d0072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:53.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.508+0000 7fe8d83b5640 1 -- 172.21.15.98:0/1916381066 shutdown_connections 2024-03-29T10:49:53.491 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.508+0000 7fe8d83b5640 1 -- 172.21.15.98:0/1916381066 >> 172.21.15.98:0/1916381066 conn(0x7fe8d0067470 msgr2=0x7fe8d00b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:53.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.509+0000 7fe8d83b5640 1 -- 172.21.15.98:0/1916381066 shutdown_connections 2024-03-29T10:49:53.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.509+0000 7fe8d83b5640 1 -- 172.21.15.98:0/1916381066 wait complete. 2024-03-29T10:49:53.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.510+0000 7fe8d83b5640 10 monclient: build_initial_monmap 2024-03-29T10:49:53.492 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.510+0000 7fe8d83b5640 10 monclient: monmap: 2024-03-29T10:49:53.492 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.510+0000 7fe8d83b5640 1 librados: starting msgr at 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.510+0000 7fe8d83b5640 1 librados: starting objecter 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.510+0000 7fe8d83b5640 1 Processor -- start 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.511+0000 7fe8d83b5640 1 -- start start 2024-03-29T10:49:53.493 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.511+0000 7fe8d83b5640 1 librados: setting wanted keys 2024-03-29T10:49:53.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.511+0000 7fe8d83b5640 1 librados: calling monclient init 2024-03-29T10:49:53.494 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.511+0000 7fe8d83b5640 10 monclient: init 2024-03-29T10:49:53.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.511+0000 7fe8d83b5640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:53.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.511+0000 7fe8d83b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:53.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.511+0000 7fe8d83b5640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d006ab70 addr v1:172.21.15.98:6789/0 2024-03-29T10:49:53.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.512+0000 7fe8d83b5640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d0141fb0 addr v1:172.21.15.189:6789/0 2024-03-29T10:49:53.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.512+0000 7fe8d83b5640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d0131e50 addr v1:172.21.15.98:6790/0 2024-03-29T10:49:53.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.512+0000 7fe8d83b5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d01dbb10 con 0x7fe8d006ab70 2024-03-29T10:49:53.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.512+0000 7fe8d83b5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d0133800 con 0x7fe8d0141fb0 2024-03-29T10:49:53.495 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.512+0000 7fe8d83b5640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d0133ae0 con 0x7fe8d0131e50 2024-03-29T10:49:53.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.512+0000 7fe8d83b5640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:53.496 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.512+0000 7fe8d83b5640 10 monclient(hunting): authenticate will time out at 1213.709200s 2024-03-29T10:49:53.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.517+0000 7fe8d5950640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe8d0141fb0 0x7fe8d01dacf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:56647/0 (socket says 172.21.15.98:56647) 2024-03-29T10:49:53.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.517+0000 7fe8d5950640 1 -- 172.21.15.98:0/525954676 learned_addr learned my addr 172.21.15.98:0/525954676 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:49:53.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.518+0000 7fe8c6ffd640 10 client.?.objecter ms_handle_connect 0x7fe8d006ab70 2024-03-29T10:49:53.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.518+0000 7fe8c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:53.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.518+0000 7fe8c6ffd640 10 client.?.objecter ms_handle_connect 0x7fe8d0141fb0 2024-03-29T10:49:53.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.518+0000 7fe8c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:53.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.519+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441554840 0 0) 0x7fe8d0133800 con 0x7fe8d0141fb0 2024-03-29T10:49:53.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.519+0000 7fe8c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:53.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.519+0000 7fe8c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:53.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.519+0000 7fe8c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:53.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.519+0000 7fe8c6ffd640 10 monclient(hunting): my global_id is 4405 2024-03-29T10:49:53.504 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.519+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b4004eb0 con 0x7fe8d0141fb0 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.519+0000 7fe8c6ffd640 10 client.?.objecter ms_handle_connect 0x7fe8d0131e50 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.519+0000 7fe8c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.519+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712258119 0 0) 0x7fe8d01dbb10 con 0x7fe8d006ab70 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.519+0000 7fe8c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.520+0000 7fe8c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.520+0000 7fe8c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.520+0000 7fe8c6ffd640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.520+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d0133800 con 0x7fe8d006ab70 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.520+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874252874 0 0) 0x7fe8d0133ae0 con 0x7fe8d0131e50 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.520+0000 7fe8c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.520+0000 7fe8c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.520+0000 7fe8c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.520+0000 7fe8c6ffd640 10 monclient(hunting): my global_id is 4430 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.520+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d01dbb10 con 0x7fe8d0131e50 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.520+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3413400546 0 0) 0x7fe8b4004eb0 con 0x7fe8d0141fb0 2024-03-29T10:49:53.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.520+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8d0133ae0 con 0x7fe8d0141fb0 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.520+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe8c0001550 con 0x7fe8d0141fb0 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:53.506 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 183088140 0 0) 0x7fe8d0133ae0 con 0x7fe8d0141fb0 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 >> v1:172.21.15.98:6790/0 conn(0x7fe8d0131e50 legacy=0x7fe8d01db400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 >> v1:172.21.15.98:6789/0 conn(0x7fe8d006ab70 legacy=0x7fe8d01341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 1 monclient: found mon.b 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0133dc0 con 0x7fe8d0141fb0 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.521+0000 7fe8c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.522+0000 7fe8d83b5640 5 monclient: authenticate success, global_id 4405 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.522+0000 7fe8d83b5640 10 monclient: _renew_subs 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.522+0000 7fe8d83b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:53.507 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.522+0000 7fe8d83b5640 1 -- 172.21.15.98:0/525954676 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d0136f70 con 0x7fe8d0141fb0 2024-03-29T10:49:53.508 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.522+0000 7fe8d83b5640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:53.508 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.522+0000 7fe8d83b5640 10 monclient: _renew_subs 2024-03-29T10:49:53.508 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.522+0000 7fe8d83b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:53.508 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.522+0000 7fe8d83b5640 1 -- 172.21.15.98:0/525954676 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d00b4240 con 0x7fe8d0141fb0 2024-03-29T10:49:53.508 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.522+0000 7fe8d83b5640 1 librados: init done 2024-03-29T10:49:53.510 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.523+0000 7fe8d83b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:53.510 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.523+0000 7fe8d83b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:53.510 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.523+0000 7fe8d83b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:53.510 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.523+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8c0004bf0 con 0x7fe8d0141fb0 2024-03-29T10:49:53.510 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.523+0000 7fe8d83b5640 1 -- 172.21.15.98:0/525954676 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d0075f10 con 0x7fe8d0141fb0 2024-03-29T10:49:53.510 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.524+0000 7fe8c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:53.510 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.526+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe8c0004f00 con 0x7fe8d0141fb0 2024-03-29T10:49:53.510 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.526+0000 7fe8c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:53.510 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.526+0000 7fe8c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:53.510 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.526+0000 7fe8c6ffd640 10 monclient: dump: 2024-03-29T10:49:53.510 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:49:53.510 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:49:53.510 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:53.511 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:49:53.511 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:53.511 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:49:53.511 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:49:53.511 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:49:53.511 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:49:53.511 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:49:53.511 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.526+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fe8c001aed0 con 0x7fe8d0141fb0 2024-03-29T10:49:53.511 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.526+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fe8c0084bb0 con 0x7fe8d0141fb0 2024-03-29T10:49:53.511 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.526+0000 7fe8c6ffd640 10 client.4405.objecter ms_dispatch 0x7fe8d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:53.511 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.526+0000 7fe8c6ffd640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:53.511 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.526+0000 7fe8c6ffd640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:53.511 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.527+0000 7fe8c6ffd640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-29T10:49:53.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.527+0000 7fe8c7fff640 4 set_mon_vals no callback set 2024-03-29T10:49:53.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.528+0000 7fe8d6151640 10 monclient: get_auth_request con 0x7fe8b406df60 auth_method 0 2024-03-29T10:49:53.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.529+0000 7fe8c6ffd640 10 client.4405.objecter ms_handle_connect 0x7fe8b406df60 2024-03-29T10:49:53.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.534+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8c0050510 con 0x7fe8d0141fb0 2024-03-29T10:49:53.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.534+0000 7fe8c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:53.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.534+0000 7fe8c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:53.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.858+0000 7fe8d83b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:53.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.858+0000 7fe8d83b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:53.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.858+0000 7fe8d83b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:49:53.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.858+0000 7fe8d83b5640 1 -- 172.21.15.98:0/525954676 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8d0137100 con 0x7fe8d0141fb0 2024-03-29T10:49:53.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.859+0000 7fe8c6ffd640 1 -- 172.21.15.98:0/525954676 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fe8c0052440 con 0x7fe8d0141fb0 2024-03-29T10:49:53.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.859+0000 7fe8c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:53.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.859+0000 7fe8c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:53.842 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:49:53.842 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:53.845 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.863+0000 7fe8d83b5640 10 librados: watch_flush enter 2024-03-29T10:49:53.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.863+0000 7fe8d83b5640 10 librados: watch_flush exit 2024-03-29T10:49:53.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.863+0000 7fe8d83b5640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-29T10:49:53.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.863+0000 7fe8d83b5640 10 client.4405.objecter successfully canceled tick 2024-03-29T10:49:53.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.863+0000 7fe8d83b5640 1 -- 172.21.15.98:0/525954676 >> v1:172.21.15.98:6816/1481799426 conn(0x7fe8b406df60 legacy=0x7fe8b4070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:53.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.863+0000 7fe8d83b5640 10 monclient: shutdown 2024-03-29T10:49:53.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.863+0000 7fe8d83b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:53.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.863+0000 7fe8d83b5640 1 -- 172.21.15.98:0/525954676 >> v1:172.21.15.189:6789/0 conn(0x7fe8d0141fb0 legacy=0x7fe8d01dacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:53.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.864+0000 7fe8d83b5640 1 -- 172.21.15.98:0/525954676 shutdown_connections 2024-03-29T10:49:53.846 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.864+0000 7fe8d83b5640 1 -- 172.21.15.98:0/525954676 >> 172.21.15.98:0/525954676 conn(0x7fe8d0067470 msgr2=0x7fe8d0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:53.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.866+0000 7fe8d83b5640 1 -- 172.21.15.98:0/525954676 shutdown_connections 2024-03-29T10:49:53.847 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.866+0000 7fe8d83b5640 1 -- 172.21.15.98:0/525954676 wait complete. 2024-03-29T10:49:53.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:49:53.867+0000 7fe8d83b5640 1 librados: shutdown 2024-03-29T10:49:53.867 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:00.674 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:00.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.851+0000 7f046e267640 10 monclient: get_monmap_and_config 2024-03-29T10:50:00.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.851+0000 7f046e267640 10 monclient: build_initial_monmap 2024-03-29T10:50:00.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.851+0000 7f046e267640 10 monclient: monmap: 2024-03-29T10:50:00.833 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:00.833 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:00.833 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:50:00.852452+0000 2024-03-29T10:50:00.834 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:50:00.852452+0000 2024-03-29T10:50:00.834 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:00.834 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:00.834 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:00.834 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:50:00.834 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:50:00.834 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:00.835 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.853+0000 7f046e267640 1 Processor -- start 2024-03-29T10:50:00.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.853+0000 7f046e267640 1 -- start start 2024-03-29T10:50:00.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.853+0000 7f046e267640 10 monclient: init 2024-03-29T10:50:00.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.854+0000 7f046e267640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:00.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.854+0000 7f046e267640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:00.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.854+0000 7f046e267640 10 monclient(hunting): picked mon.noname-c con 0x7f04680b4820 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:00.838 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.854+0000 7f046e267640 10 monclient(hunting): picked mon.noname-a con 0x7f04680a9ce0 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:00.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.854+0000 7f046e267640 10 monclient(hunting): picked mon.noname-b con 0x7f04680b3cd0 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:00.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.854+0000 7f046e267640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04680b4180 con 0x7f04680a9ce0 2024-03-29T10:50:00.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.854+0000 7f046e267640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04680aaf60 con 0x7f04680b4820 2024-03-29T10:50:00.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.854+0000 7f046e267640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046813c200 con 0x7f04680b3cd0 2024-03-29T10:50:00.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.854+0000 7f046e267640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:00.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.854+0000 7f046e267640 10 monclient(hunting): authenticate will time out at 1221.051637s 2024-03-29T10:50:00.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.858+0000 7f046d265640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f04680b4820 0x7f046813baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:44157/0 (socket says 172.21.15.98:44157) 2024-03-29T10:50:00.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.858+0000 7f046d265640 1 -- 172.21.15.98:0/735368242 learned_addr learned my addr 172.21.15.98:0/735368242 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:00.841 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.860+0000 7f045ffff640 1 -- 172.21.15.98:0/735368242 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243959449 0 0) 0x7f04680aaf60 con 0x7f04680b4820 2024-03-29T10:50:00.842 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.860+0000 7f045ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:00.849 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.860+0000 7f045ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:00.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.862+0000 7f045ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:00.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.862+0000 7f045ffff640 10 monclient(hunting): my global_id is 4408 2024-03-29T10:50:00.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.862+0000 7f045ffff640 1 -- 172.21.15.98:0/735368242 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0454003520 con 0x7f04680b4820 2024-03-29T10:50:00.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.862+0000 7f045ffff640 1 -- 172.21.15.98:0/735368242 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067297193 0 0) 0x7f046813c200 con 0x7f04680b3cd0 2024-03-29T10:50:00.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.862+0000 7f045ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:00.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.862+0000 7f045ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:00.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.862+0000 7f045ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:00.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.862+0000 7f045ffff640 10 monclient(hunting): my global_id is 4433 2024-03-29T10:50:00.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.862+0000 7f045ffff640 1 -- 172.21.15.98:0/735368242 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04680aaf60 con 0x7f04680b3cd0 2024-03-29T10:50:00.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.863+0000 7f046d265640 1 -- 172.21.15.98:0/735368242 >> v1:172.21.15.189:6789/0 conn(0x7f04680b4820 legacy=0x7f046813baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T10:50:00.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.863+0000 7f046d265640 1 -- 172.21.15.98:0/735368242 >> v1:172.21.15.189:6789/0 conn(0x7f04680b4820 legacy=0x7f046813baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:50:00.850 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.863+0000 7f046d265640 1 --1- 172.21.15.98:0/735368242 >> v1:172.21.15.189:6789/0 conn(0x7f04680b4820 0x7f046813baf0 :-1 s=OPENED pgs=665784 cs=1 l=1).handle_message read tag failed 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.863+0000 7f046d265640 1 --1- 172.21.15.98:0/735368242 >> v1:172.21.15.189:6789/0 conn(0x7f04680b4820 0x7f046813baf0 :-1 s=OPENED pgs=665784 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.863+0000 7f045ffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.189:6789/0 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 1 -- 172.21.15.98:0/735368242 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 994762758 0 0) 0x7f04680aaf60 con 0x7f04680b3cd0 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 1 -- 172.21.15.98:0/735368242 >> v1:172.21.15.189:6789/0 conn(0x7f04680b4820 legacy=0x7f046813baf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 1 -- 172.21.15.98:0/735368242 >> v1:172.21.15.98:6789/0 conn(0x7f04680a9ce0 legacy=0x7f0468145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 1 monclient: found mon.noname-b 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 1 -- 172.21.15.98:0/735368242 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0468145e80 con 0x7f04680b3cd0 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 10 monclient: _finish_auth 0 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:00.851 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 1 -- 172.21.15.98:0/735368242 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f0460001670 con 0x7f04680b3cd0 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.864+0000 7f045ffff640 10 monclient: dump: 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.865+0000 7f046e267640 5 monclient: authenticate success, global_id 4433 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.865+0000 7f046e267640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:00.852 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.865+0000 7f045ffff640 1 -- 172.21.15.98:0/735368242 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04600017d0 con 0x7f04680b3cd0 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.865+0000 7f045ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.865+0000 7f046e267640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.865+0000 7f046e267640 4 set_mon_vals no callback set 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.865+0000 7f046e267640 10 monclient: shutdown 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.866+0000 7f046e267640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.866+0000 7f046e267640 1 -- 172.21.15.98:0/735368242 >> v1:172.21.15.98:6790/0 conn(0x7f04680b3cd0 legacy=0x7f046814d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.866+0000 7f046e267640 1 -- 172.21.15.98:0/735368242 shutdown_connections 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.866+0000 7f046e267640 1 -- 172.21.15.98:0/735368242 >> 172.21.15.98:0/735368242 conn(0x7f046806d070 msgr2=0x7f0468066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.866+0000 7f046e267640 1 -- 172.21.15.98:0/735368242 shutdown_connections 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.866+0000 7f046e267640 1 -- 172.21.15.98:0/735368242 wait complete. 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.867+0000 7f046e267640 10 monclient: build_initial_monmap 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.867+0000 7f046e267640 10 monclient: monmap: 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:50:00.853 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:50:00.854 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:00.854 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:00.854 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:00.854 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:50:00.854 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:50:00.854 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:00.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.867+0000 7f046e267640 1 librados: starting msgr at 2024-03-29T10:50:00.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.868+0000 7f046e267640 1 librados: starting objecter 2024-03-29T10:50:00.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.868+0000 7f046e267640 1 Processor -- start 2024-03-29T10:50:00.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.868+0000 7f046e267640 1 -- start start 2024-03-29T10:50:00.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.868+0000 7f046e267640 1 librados: setting wanted keys 2024-03-29T10:50:00.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.868+0000 7f046e267640 1 librados: calling monclient init 2024-03-29T10:50:00.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.868+0000 7f046e267640 10 monclient: init 2024-03-29T10:50:00.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.869+0000 7f046e267640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:00.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.869+0000 7f046e267640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:00.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.869+0000 7f046e267640 10 monclient(hunting): picked mon.noname-c con 0x7f04680a9ce0 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:00.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.869+0000 7f046e267640 10 monclient(hunting): picked mon.noname-a con 0x7f04680b3cd0 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:00.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.869+0000 7f046e267640 10 monclient(hunting): picked mon.noname-b con 0x7f04680b4820 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:00.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.869+0000 7f046e267640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04681dec80 con 0x7f04680b3cd0 2024-03-29T10:50:00.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.870+0000 7f046e267640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04681df8b0 con 0x7f04680b4820 2024-03-29T10:50:00.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.870+0000 7f046e267640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04681dfa40 con 0x7f04680a9ce0 2024-03-29T10:50:00.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.870+0000 7f046e267640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:00.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.870+0000 7f046e267640 10 monclient(hunting): authenticate will time out at 1221.067188s 2024-03-29T10:50:00.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.876+0000 7f046d265640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f04680a9ce0 0x7f04681df1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:36207/0 (socket says 172.21.15.98:36207) 2024-03-29T10:50:00.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.876+0000 7f046d265640 1 -- 172.21.15.98:0/1514940270 learned_addr learned my addr 172.21.15.98:0/1514940270 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:00.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.877+0000 7f045dffb640 10 client.?.objecter ms_handle_connect 0x7f04680b4820 2024-03-29T10:50:00.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:00.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 client.?.objecter ms_handle_connect 0x7f04680b3cd0 2024-03-29T10:50:00.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:00.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161758702 0 0) 0x7f04681df8b0 con 0x7f04680b4820 2024-03-29T10:50:00.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:00.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:00.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:00.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 monclient(hunting): my global_id is 4411 2024-03-29T10:50:00.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f044c004eb0 con 0x7f04680b4820 2024-03-29T10:50:00.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 client.?.objecter ms_handle_connect 0x7f04680a9ce0 2024-03-29T10:50:00.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:00.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484690463 0 0) 0x7f04681dec80 con 0x7f04680b3cd0 2024-03-29T10:50:00.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:00.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:00.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 monclient(hunting): my global_id is 4410 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04681df8b0 con 0x7f04680b3cd0 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101538948 0 0) 0x7f0454003520 con 0x7f04680a9ce0 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 10 monclient(hunting): my global_id is 4436 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.878+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04681dec80 con 0x7f04680a9ce0 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 502051484 0 0) 0x7f04681dec80 con 0x7f04680a9ce0 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0454003520 con 0x7f04680a9ce0 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f0458001860 con 0x7f04680a9ce0 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:00.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 10 monclient(hunting): dump: 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3989439662 0 0) 0x7f04681df8b0 con 0x7f04680b3cd0 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04681dec80 con 0x7f04680b3cd0 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f0464001a60 con 0x7f04680b3cd0 2024-03-29T10:50:00.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 10 monclient(hunting): dump: 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3799692011 0 0) 0x7f0454003520 con 0x7f04680a9ce0 2024-03-29T10:50:00.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 >> v1:172.21.15.189:6789/0 conn(0x7f04680b4820 legacy=0x7f04681e6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 >> v1:172.21.15.98:6789/0 conn(0x7f04680b3cd0 legacy=0x7f04680aceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 1 monclient: found mon.c 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04680ac330 con 0x7f04680a9ce0 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.879+0000 7f045dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f045dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f046e267640 5 monclient: authenticate success, global_id 4436 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f046e267640 10 monclient: _renew_subs 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f046e267640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f046e267640 1 -- 172.21.15.98:0/1514940270 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0468070830 con 0x7f04680a9ce0 2024-03-29T10:50:00.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f046e267640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f046e267640 10 monclient: _renew_subs 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f046e267640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f046e267640 1 -- 172.21.15.98:0/1514940270 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04680ad7a0 con 0x7f04680a9ce0 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f046e267640 1 librados: init done 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0458001c70 con 0x7f04680a9ce0 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f045dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f0458004980 con 0x7f04680a9ce0 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f045effd640 4 set_mon_vals no callback set 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.880+0000 7f045dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.881+0000 7f045dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.881+0000 7f045dffb640 10 monclient: dump: 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:00.868 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:00.869 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:00.869 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:00.869 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:00.869 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:00.869 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:00.869 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:00.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.882+0000 7f046e267640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:00.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.882+0000 7f046e267640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:00.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.883+0000 7f046e267640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:00.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.883+0000 7f046e267640 1 -- 172.21.15.98:0/1514940270 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0448005080 con 0x7f04680a9ce0 2024-03-29T10:50:00.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.886+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f045801a950 con 0x7f04680a9ce0 2024-03-29T10:50:00.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.887+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f0458084830 con 0x7f04680a9ce0 2024-03-29T10:50:00.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.887+0000 7f045dffb640 10 client.4436.objecter ms_dispatch 0x7f0468000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:00.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.887+0000 7f045dffb640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:00.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.887+0000 7f045dffb640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:00.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.887+0000 7f045dffb640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T10:50:00.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.887+0000 7f046ca64640 10 monclient: get_auth_request con 0x7f044c06e050 auth_method 0 2024-03-29T10:50:00.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.888+0000 7f045dffb640 10 client.4436.objecter ms_handle_connect 0x7f044c06e050 2024-03-29T10:50:00.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.897+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0458050110 con 0x7f04680a9ce0 2024-03-29T10:50:00.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.897+0000 7f045dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:00.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:00.897+0000 7f045dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:01.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.251+0000 7f046e267640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:01.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.251+0000 7f046e267640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:01.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.251+0000 7f046e267640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:01.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.251+0000 7f046e267640 1 -- 172.21.15.98:0/1514940270 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0448005330 con 0x7f04680a9ce0 2024-03-29T10:50:01.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.252+0000 7f045dffb640 1 -- 172.21.15.98:0/1514940270 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f0458052040 con 0x7f04680a9ce0 2024-03-29T10:50:01.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.252+0000 7f045dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:01.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.252+0000 7f045dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:01.235 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:50:01.235 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:01.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.257+0000 7f046e267640 10 librados: watch_flush enter 2024-03-29T10:50:01.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.257+0000 7f046e267640 10 librados: watch_flush exit 2024-03-29T10:50:01.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.257+0000 7f046e267640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T10:50:01.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.257+0000 7f046e267640 10 client.4436.objecter successfully canceled tick 2024-03-29T10:50:01.239 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.257+0000 7f046e267640 1 -- 172.21.15.98:0/1514940270 >> v1:172.21.15.98:6816/1481799426 conn(0x7f044c06e050 legacy=0x7f044c070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:01.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.257+0000 7f046e267640 10 monclient: shutdown 2024-03-29T10:50:01.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.257+0000 7f046e267640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:01.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.257+0000 7f046e267640 1 -- 172.21.15.98:0/1514940270 >> v1:172.21.15.98:6790/0 conn(0x7f04680a9ce0 legacy=0x7f04681df1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:01.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.257+0000 7f046e267640 1 -- 172.21.15.98:0/1514940270 shutdown_connections 2024-03-29T10:50:01.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.257+0000 7f046e267640 1 -- 172.21.15.98:0/1514940270 >> 172.21.15.98:0/1514940270 conn(0x7f046806d070 msgr2=0x7f046813fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:01.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.257+0000 7f046e267640 1 -- 172.21.15.98:0/1514940270 shutdown_connections 2024-03-29T10:50:01.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.258+0000 7f046e267640 1 -- 172.21.15.98:0/1514940270 wait complete. 2024-03-29T10:50:01.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:01.258+0000 7f046e267640 1 librados: shutdown 2024-03-29T10:50:01.260 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:08.067 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:08.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.227+0000 7fdec17c6640 10 monclient: get_monmap_and_config 2024-03-29T10:50:08.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.227+0000 7fdec17c6640 10 monclient: build_initial_monmap 2024-03-29T10:50:08.210 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.227+0000 7fdec17c6640 10 monclient: monmap: 2024-03-29T10:50:08.210 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:08.210 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:08.210 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:50:08.229083+0000 2024-03-29T10:50:08.210 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:50:08.229083+0000 2024-03-29T10:50:08.210 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:08.210 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:08.210 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:08.210 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:50:08.211 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:50:08.211 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:08.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.228+0000 7fdec17c6640 1 Processor -- start 2024-03-29T10:50:08.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.228+0000 7fdec17c6640 1 -- start start 2024-03-29T10:50:08.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.228+0000 7fdec17c6640 10 monclient: init 2024-03-29T10:50:08.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.229+0000 7fdec17c6640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:08.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.229+0000 7fdec17c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:08.215 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.229+0000 7fdec17c6640 10 monclient(hunting): picked mon.noname-b con 0x7fdebc0b4890 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.229+0000 7fdec17c6640 10 monclient(hunting): picked mon.noname-c con 0x7fdebc13c040 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.229+0000 7fdec17c6640 10 monclient(hunting): picked mon.noname-a con 0x7fdebc1427e0 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.229+0000 7fdec17c6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdebc0b42f0 con 0x7fdebc1427e0 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.229+0000 7fdec17c6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdebc0aa7a0 con 0x7fdebc13c040 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.229+0000 7fdec17c6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdebc0aaa10 con 0x7fdebc0b4890 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.229+0000 7fdec17c6640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.229+0000 7fdeb27fc640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdebc13c040 0x7fdebc1420d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:44722/0 (socket says 172.21.15.98:44722) 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.229+0000 7fdeb27fc640 1 -- 172.21.15.98:0/1111887007 learned_addr learned my addr 172.21.15.98:0/1111887007 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.229+0000 7fdec17c6640 10 monclient(hunting): authenticate will time out at 1228.427838s 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 1 -- 172.21.15.98:0/1111887007 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668677613 0 0) 0x7fdebc0aa7a0 con 0x7fdebc13c040 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient(hunting): my global_id is 4414 2024-03-29T10:50:08.216 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 1 -- 172.21.15.98:0/1111887007 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdea8003520 con 0x7fdebc13c040 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 1 -- 172.21.15.98:0/1111887007 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277791658 0 0) 0x7fdebc0b42f0 con 0x7fdebc1427e0 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient(hunting): my global_id is 4413 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 1 -- 172.21.15.98:0/1111887007 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdebc0aa7a0 con 0x7fdebc1427e0 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 1 -- 172.21.15.98:0/1111887007 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2993134342 0 0) 0x7fdea8003520 con 0x7fdebc13c040 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 1 -- 172.21.15.98:0/1111887007 >> v1:172.21.15.98:6790/0 conn(0x7fdebc0b4890 legacy=0x7fdebc13b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 1 -- 172.21.15.98:0/1111887007 >> v1:172.21.15.98:6789/0 conn(0x7fdebc1427e0 legacy=0x7fdebc14d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 1 monclient: found mon.noname-c 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:50:08.217 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 1 -- 172.21.15.98:0/1111887007 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdebc0aacf0 con 0x7fdebc13c040 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 1 -- 172.21.15.98:0/1111887007 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fdea4001690 con 0x7fdebc13c040 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdeba7fc640 10 monclient: dump: 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:08.218 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdec17c6640 5 monclient: authenticate success, global_id 4414 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.230+0000 7fdec17c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.231+0000 7fdeba7fc640 1 -- 172.21.15.98:0/1111887007 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdea40017f0 con 0x7fdebc13c040 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.231+0000 7fdeba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.231+0000 7fdec17c6640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.231+0000 7fdec17c6640 4 set_mon_vals no callback set 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.231+0000 7fdeba7fc640 1 -- 172.21.15.98:0/1111887007 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fdea4001b80 con 0x7fdebc13c040 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.231+0000 7fdeba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.231+0000 7fdeba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.231+0000 7fdeba7fc640 10 monclient: dump: 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:08.219 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.232+0000 7fdec17c6640 10 monclient: shutdown 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.232+0000 7fdec17c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.232+0000 7fdec17c6640 1 -- 172.21.15.98:0/1111887007 >> v1:172.21.15.189:6789/0 conn(0x7fdebc13c040 legacy=0x7fdebc1420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.232+0000 7fdec17c6640 1 -- 172.21.15.98:0/1111887007 shutdown_connections 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.232+0000 7fdec17c6640 1 -- 172.21.15.98:0/1111887007 >> 172.21.15.98:0/1111887007 conn(0x7fdebc06d180 msgr2=0x7fdebc066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:08.220 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.232+0000 7fdec17c6640 1 -- 172.21.15.98:0/1111887007 shutdown_connections 2024-03-29T10:50:08.221 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.232+0000 7fdec17c6640 1 -- 172.21.15.98:0/1111887007 wait complete. 2024-03-29T10:50:08.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.233+0000 7fdec17c6640 10 monclient: build_initial_monmap 2024-03-29T10:50:08.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.233+0000 7fdec17c6640 10 monclient: monmap: 2024-03-29T10:50:08.227 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:08.227 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:08.227 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:50:08.227 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:50:08.227 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:08.227 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:08.227 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:08.227 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:50:08.227 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:50:08.227 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:08.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.233+0000 7fdec17c6640 1 librados: starting msgr at 2024-03-29T10:50:08.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.233+0000 7fdec17c6640 1 librados: starting objecter 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.234+0000 7fdec17c6640 1 Processor -- start 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.234+0000 7fdec17c6640 1 -- start start 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.234+0000 7fdec17c6640 1 librados: setting wanted keys 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.234+0000 7fdec17c6640 1 librados: calling monclient init 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.234+0000 7fdec17c6640 10 monclient: init 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.235+0000 7fdec17c6640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.235+0000 7fdec17c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.235+0000 7fdec17c6640 10 monclient(hunting): picked mon.noname-b con 0x7fdebc0b4890 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.235+0000 7fdebaffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdebc0b4890 0x7fdebc145a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:45114/0 (socket says 172.21.15.98:45114) 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.235+0000 7fdebaffd640 1 -- 172.21.15.98:0/4036128753 learned_addr learned my addr 172.21.15.98:0/4036128753 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.235+0000 7fdec17c6640 10 monclient(hunting): picked mon.noname-a con 0x7fdebc13c040 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.236+0000 7fdeb3fff640 10 client.?.objecter ms_handle_connect 0x7fdebc0b4890 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.236+0000 7fdec17c6640 10 monclient(hunting): picked mon.noname-c con 0x7fdebc1427e0 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.236+0000 7fdeb3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:08.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.236+0000 7fdeb3fff640 10 client.?.objecter ms_handle_connect 0x7fdebc13c040 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.236+0000 7fdeb3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.236+0000 7fdec17c6640 1 -- 172.21.15.98:0/4036128753 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdebc1d71e0 con 0x7fdebc13c040 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.236+0000 7fdec17c6640 1 -- 172.21.15.98:0/4036128753 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdebc0acdf0 con 0x7fdebc0b4890 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.236+0000 7fdec17c6640 1 -- 172.21.15.98:0/4036128753 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdebc1d6e70 con 0x7fdebc1427e0 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.236+0000 7fdec17c6640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.236+0000 7fdec17c6640 10 monclient(hunting): authenticate will time out at 1228.434739s 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.236+0000 7fdeb3fff640 10 client.?.objecter ms_handle_connect 0x7fdebc1427e0 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.236+0000 7fdeb3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074580968 0 0) 0x7fdebc1d71e0 con 0x7fdebc13c040 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): my global_id is 4416 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde9c004eb0 con 0x7fdebc13c040 2024-03-29T10:50:08.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10657986 0 0) 0x7fdebc1d6e70 con 0x7fdebc1427e0 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdebc1d71e0 con 0x7fdebc1427e0 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760217224 0 0) 0x7fdebc0acdf0 con 0x7fdebc0b4890 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): my global_id is 4417 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdebc1d6e70 con 0x7fdebc0b4890 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3517828719 0 0) 0x7fdebc1d71e0 con 0x7fdebc1427e0 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdebc0acdf0 con 0x7fdebc1427e0 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fdeb4001b50 con 0x7fdebc1427e0 2024-03-29T10:50:08.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:08.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:08.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): dump: 2024-03-29T10:50:08.231 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:08.231 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:08.231 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:08.231 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:08.231 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:08.231 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:08.231 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:08.231 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:08.231 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:08.231 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:08.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.237+0000 7fdeb3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 677759292 0 0) 0x7fdebc0acdf0 con 0x7fdebc1427e0 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 >> v1:172.21.15.189:6789/0 conn(0x7fdebc0b4890 legacy=0x7fdebc145a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 >> v1:172.21.15.98:6789/0 conn(0x7fdebc13c040 legacy=0x7fdebc1d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 1 monclient: found mon.c 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdebc1d7d80 con 0x7fdebc1427e0 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 10 monclient: _finish_auth 0 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdeb40020c0 con 0x7fdebc1427e0 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fdeb4003b30 con 0x7fdebc1427e0 2024-03-29T10:50:08.232 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb3fff640 10 monclient: dump: 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdeb9ffb640 4 set_mon_vals no callback set 2024-03-29T10:50:08.233 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdec17c6640 5 monclient: authenticate success, global_id 4442 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdec17c6640 10 monclient: _renew_subs 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdec17c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.238+0000 7fdec17c6640 1 -- 172.21.15.98:0/4036128753 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdebc138450 con 0x7fdebc1427e0 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.239+0000 7fdec17c6640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.239+0000 7fdec17c6640 10 monclient: _renew_subs 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.239+0000 7fdec17c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.239+0000 7fdec17c6640 1 -- 172.21.15.98:0/4036128753 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdebc1432e0 con 0x7fdebc1427e0 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.239+0000 7fdec17c6640 1 librados: init done 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.240+0000 7fdeb17fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.240+0000 7fdeb17fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.240+0000 7fdeb17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.240+0000 7fdeb17fa640 1 -- 172.21.15.98:0/4036128753 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde94005080 con 0x7fdebc1427e0 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.241+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fdeb4019b00 con 0x7fdebc1427e0 2024-03-29T10:50:08.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.242+0000 7fdebaffd640 10 monclient: get_auth_request con 0x7fde9c06de80 auth_method 0 2024-03-29T10:50:08.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.242+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fdeb40839d0 con 0x7fdebc1427e0 2024-03-29T10:50:08.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.242+0000 7fdeb3fff640 10 client.4442.objecter ms_dispatch 0x7fdebc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:08.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.242+0000 7fdeb3fff640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:08.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.242+0000 7fdeb3fff640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:08.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.243+0000 7fdeb3fff640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-29T10:50:08.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.243+0000 7fdeb3fff640 10 client.4442.objecter ms_handle_connect 0x7fde9c06de80 2024-03-29T10:50:08.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.249+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdeb404f2b0 con 0x7fdebc1427e0 2024-03-29T10:50:08.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.249+0000 7fdeb3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:08.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.249+0000 7fdeb3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:08.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.650+0000 7fdeb17fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.650+0000 7fdeb17fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.650+0000 7fdeb17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.651+0000 7fdeb17fa640 1 -- 172.21.15.98:0/4036128753 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde94005330 con 0x7fdebc1427e0 2024-03-29T10:50:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.651+0000 7fdeb3fff640 1 -- 172.21.15.98:0/4036128753 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fdeb40511e0 con 0x7fdebc1427e0 2024-03-29T10:50:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.652+0000 7fdeb3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:08.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.652+0000 7fdeb3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:08.635 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:50:08.635 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:08.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.655+0000 7fdeb17fa640 10 librados: watch_flush enter 2024-03-29T10:50:08.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.655+0000 7fdeb17fa640 10 librados: watch_flush exit 2024-03-29T10:50:08.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.655+0000 7fdeb17fa640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-29T10:50:08.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.655+0000 7fdeb17fa640 10 client.4442.objecter successfully canceled tick 2024-03-29T10:50:08.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.656+0000 7fdeb17fa640 1 -- 172.21.15.98:0/4036128753 >> v1:172.21.15.98:6816/1481799426 conn(0x7fde9c06de80 legacy=0x7fde9c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:08.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.656+0000 7fdeb17fa640 10 monclient: shutdown 2024-03-29T10:50:08.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.656+0000 7fdeb17fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:08.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.656+0000 7fdeb17fa640 1 -- 172.21.15.98:0/4036128753 >> v1:172.21.15.98:6790/0 conn(0x7fdebc1427e0 legacy=0x7fdebc137d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:08.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.656+0000 7fdeb17fa640 1 -- 172.21.15.98:0/4036128753 shutdown_connections 2024-03-29T10:50:08.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.656+0000 7fdeb17fa640 1 -- 172.21.15.98:0/4036128753 >> 172.21.15.98:0/4036128753 conn(0x7fdebc06d180 msgr2=0x7fdebc0b5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:08.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.656+0000 7fdeb17fa640 1 -- 172.21.15.98:0/4036128753 shutdown_connections 2024-03-29T10:50:08.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.656+0000 7fdeb17fa640 1 -- 172.21.15.98:0/4036128753 wait complete. 2024-03-29T10:50:08.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:08.657+0000 7fdeb17fa640 1 librados: shutdown 2024-03-29T10:50:08.655 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:15.462 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:15.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.630+0000 7fe21dab5640 10 monclient: get_monmap_and_config 2024-03-29T10:50:15.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.630+0000 7fe21dab5640 10 monclient: build_initial_monmap 2024-03-29T10:50:15.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.630+0000 7fe21dab5640 10 monclient: monmap: 2024-03-29T10:50:15.614 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:15.614 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:15.614 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:50:15.632432+0000 2024-03-29T10:50:15.614 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:50:15.632432+0000 2024-03-29T10:50:15.614 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:15.614 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:15.614 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:15.614 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:50:15.614 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:50:15.614 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:15.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.632+0000 7fe21dab5640 1 Processor -- start 2024-03-29T10:50:15.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.632+0000 7fe21dab5640 1 -- start start 2024-03-29T10:50:15.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.632+0000 7fe21dab5640 10 monclient: init 2024-03-29T10:50:15.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.633+0000 7fe21dab5640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:15.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.633+0000 7fe21dab5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:15.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.633+0000 7fe21dab5640 10 monclient(hunting): picked mon.noname-c con 0x7fe2180b1f60 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:15.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.633+0000 7fe21dab5640 10 monclient(hunting): picked mon.noname-b con 0x7fe218071eb0 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:15.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.633+0000 7fe21dab5640 10 monclient(hunting): picked mon.noname-a con 0x7fe218143d10 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:15.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.634+0000 7fe21dab5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21806ef80 con 0x7fe218143d10 2024-03-29T10:50:15.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.634+0000 7fe21dab5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21806f260 con 0x7fe2180b1f60 2024-03-29T10:50:15.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.634+0000 7fe21dab5640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2180b40c0 con 0x7fe218071eb0 2024-03-29T10:50:15.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.634+0000 7fe21dab5640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:15.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.634+0000 7fe21dab5640 10 monclient(hunting): authenticate will time out at 1235.832790s 2024-03-29T10:50:15.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.635+0000 7fe2177fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe2180b1f60 0x7fe218143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:40488/0 (socket says 172.21.15.98:40488) 2024-03-29T10:50:15.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.635+0000 7fe2177fe640 1 -- 172.21.15.98:0/3969379850 learned_addr learned my addr 172.21.15.98:0/3969379850 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:15.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.637+0000 7fe2167fc640 1 -- 172.21.15.98:0/3969379850 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249217191 0 0) 0x7fe21806ef80 con 0x7fe218143d10 2024-03-29T10:50:15.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.637+0000 7fe2167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:15.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.637+0000 7fe2167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:15.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.637+0000 7fe2167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:15.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.637+0000 7fe2167fc640 10 monclient(hunting): my global_id is 4419 2024-03-29T10:50:15.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.637+0000 7fe2167fc640 1 -- 172.21.15.98:0/3969379850 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe200003520 con 0x7fe218143d10 2024-03-29T10:50:15.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.637+0000 7fe2167fc640 1 -- 172.21.15.98:0/3969379850 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948703534 0 0) 0x7fe21806f260 con 0x7fe2180b1f60 2024-03-29T10:50:15.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.637+0000 7fe2167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:15.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.637+0000 7fe2167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:15.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.637+0000 7fe2167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:15.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.637+0000 7fe2167fc640 10 monclient(hunting): my global_id is 4420 2024-03-29T10:50:15.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.637+0000 7fe2167fc640 1 -- 172.21.15.98:0/3969379850 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe21806ef80 con 0x7fe2180b1f60 2024-03-29T10:50:15.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 1 -- 172.21.15.98:0/3969379850 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412070876 0 0) 0x7fe2180b40c0 con 0x7fe218071eb0 2024-03-29T10:50:15.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:15.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:15.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:15.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:50:15.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 1 -- 172.21.15.98:0/3969379850 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe21806f260 con 0x7fe218071eb0 2024-03-29T10:50:15.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 1 -- 172.21.15.98:0/3969379850 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 249959130 0 0) 0x7fe200003520 con 0x7fe218143d10 2024-03-29T10:50:15.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 1 -- 172.21.15.98:0/3969379850 >> v1:172.21.15.98:6790/0 conn(0x7fe218071eb0 legacy=0x7fe218146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:15.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 1 -- 172.21.15.98:0/3969379850 >> v1:172.21.15.189:6789/0 conn(0x7fe2180b1f60 legacy=0x7fe218143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:15.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:15.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 1 monclient: found mon.noname-a 2024-03-29T10:50:15.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:15.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:50:15.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 1 -- 172.21.15.98:0/3969379850 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe218155d70 con 0x7fe218143d10 2024-03-29T10:50:15.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe21dab5640 5 monclient: authenticate success, global_id 4419 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe21dab5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 1 -- 172.21.15.98:0/3969379850 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe208001690 con 0x7fe218143d10 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient: dump: 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:15.627 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe21dab5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 1 -- 172.21.15.98:0/3969379850 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2080018b0 con 0x7fe218143d10 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe21dab5640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe21dab5640 4 set_mon_vals no callback set 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 1 -- 172.21.15.98:0/3969379850 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe208001c40 con 0x7fe218143d10 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.638+0000 7fe2167fc640 10 monclient: dump: 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:15.628 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.639+0000 7fe21dab5640 10 monclient: shutdown 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.639+0000 7fe21dab5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.639+0000 7fe21dab5640 1 -- 172.21.15.98:0/3969379850 >> v1:172.21.15.98:6789/0 conn(0x7fe218143d10 legacy=0x7fe218146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.639+0000 7fe21dab5640 1 -- 172.21.15.98:0/3969379850 shutdown_connections 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.639+0000 7fe21dab5640 1 -- 172.21.15.98:0/3969379850 >> 172.21.15.98:0/3969379850 conn(0x7fe218066a00 msgr2=0x7fe218066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.639+0000 7fe21dab5640 1 -- 172.21.15.98:0/3969379850 shutdown_connections 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.639+0000 7fe21dab5640 1 -- 172.21.15.98:0/3969379850 wait complete. 2024-03-29T10:50:15.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.640+0000 7fe21dab5640 10 monclient: build_initial_monmap 2024-03-29T10:50:15.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.640+0000 7fe21dab5640 10 monclient: monmap: 2024-03-29T10:50:15.632 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:15.632 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:15.632 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:50:15.632 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:50:15.632 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.641+0000 7fe21dab5640 1 librados: starting msgr at 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.641+0000 7fe21dab5640 1 librados: starting objecter 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.641+0000 7fe21dab5640 1 Processor -- start 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.641+0000 7fe21dab5640 1 -- start start 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.641+0000 7fe21dab5640 1 librados: setting wanted keys 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.641+0000 7fe21dab5640 1 librados: calling monclient init 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.641+0000 7fe21dab5640 10 monclient: init 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.642+0000 7fe21dab5640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:15.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.642+0000 7fe21dab5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.642+0000 7fe21dab5640 10 monclient(hunting): picked mon.noname-b con 0x7fe2180b1f60 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.642+0000 7fe21dab5640 10 monclient(hunting): picked mon.noname-a con 0x7fe218143d10 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.642+0000 7fe21dab5640 10 monclient(hunting): picked mon.noname-c con 0x7fe2180ab290 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.642+0000 7fe21dab5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2181457b0 con 0x7fe218143d10 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.642+0000 7fe21dab5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2180ac9e0 con 0x7fe2180b1f60 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.642+0000 7fe21dab5640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2180ab900 con 0x7fe2180ab290 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.642+0000 7fe21dab5640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.642+0000 7fe21dab5640 10 monclient(hunting): authenticate will time out at 1235.840612s 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.645+0000 7fe217fff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fe2180ab290 0x7fe2181ffc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:52017/0 (socket says 172.21.15.98:52017) 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.645+0000 7fe217fff640 1 -- 172.21.15.98:0/348030057 learned_addr learned my addr 172.21.15.98:0/348030057 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.645+0000 7fe1fffff640 10 client.?.objecter ms_handle_connect 0x7fe2180ab290 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.645+0000 7fe1fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.646+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820472157 0 0) 0x7fe2180ab900 con 0x7fe2180ab290 2024-03-29T10:50:15.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.646+0000 7fe1fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.646+0000 7fe1fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.646+0000 7fe1fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.646+0000 7fe1fffff640 10 monclient(hunting): my global_id is 4448 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.646+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe204004eb0 con 0x7fe2180ab290 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.646+0000 7fe1fffff640 10 client.?.objecter ms_handle_connect 0x7fe2180b1f60 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.646+0000 7fe1fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718878608 0 0) 0x7fe2180ac9e0 con 0x7fe2180b1f60 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 10 monclient(hunting): my global_id is 4423 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2180ab900 con 0x7fe2180b1f60 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2276220718 0 0) 0x7fe204004eb0 con 0x7fe2180ab290 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2180ac9e0 con 0x7fe2180ab290 2024-03-29T10:50:15.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe208001bc0 con 0x7fe2180ab290 2024-03-29T10:50:15.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:15.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:15.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 10 monclient(hunting): dump: 2024-03-29T10:50:15.636 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:15.636 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:15.636 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:15.636 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:15.636 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:15.636 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:15.636 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:15.636 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:15.636 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:15.636 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:15.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:15.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 105185988 0 0) 0x7fe2180ac9e0 con 0x7fe2180ab290 2024-03-29T10:50:15.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 >> v1:172.21.15.189:6789/0 conn(0x7fe2180b1f60 legacy=0x7fe2180aab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:15.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.647+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 >> v1:172.21.15.98:6789/0 conn(0x7fe218143d10 legacy=0x7fe218138ee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:50:15.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe1fffff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:15.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe1fffff640 1 monclient: found mon.c 2024-03-29T10:50:15.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe1fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe1fffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2180b15e0 con 0x7fe2180ab290 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe1fffff640 10 monclient: _finish_auth 0 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe1fffff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe1fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe21dab5640 5 monclient: authenticate success, global_id 4448 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe21dab5640 10 monclient: _renew_subs 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe21dab5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe21dab5640 1 -- 172.21.15.98:0/348030057 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2180aa310 con 0x7fe2180ab290 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe21dab5640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe21dab5640 10 monclient: _renew_subs 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe21dab5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe21dab5640 1 -- 172.21.15.98:0/348030057 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2181397d0 con 0x7fe2180ab290 2024-03-29T10:50:15.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe21dab5640 1 librados: init done 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe208001d20 con 0x7fe2180ab290 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe1fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.648+0000 7fe2157fa640 4 set_mon_vals no callback set 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.649+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe2080038a0 con 0x7fe2180ab290 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.649+0000 7fe1fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.649+0000 7fe1fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.649+0000 7fe1fffff640 10 monclient: dump: 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:15.641 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.651+0000 7fe21dab5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.651+0000 7fe21dab5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.651+0000 7fe21dab5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.651+0000 7fe21dab5640 1 -- 172.21.15.98:0/348030057 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f8005080 con 0x7fe2180ab290 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.653+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fe208003a00 con 0x7fe2180ab290 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.654+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fe2080848f0 con 0x7fe2180ab290 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.654+0000 7fe1fffff640 10 client.4448.objecter ms_dispatch 0x7fe218000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.654+0000 7fe1fffff640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.654+0000 7fe1fffff640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.654+0000 7fe1fffff640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.654+0000 7fe2177fe640 10 monclient: get_auth_request con 0x7fe20406dde0 auth_method 0 2024-03-29T10:50:15.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.655+0000 7fe1fffff640 10 client.4448.objecter ms_handle_connect 0x7fe20406dde0 2024-03-29T10:50:15.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.659+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe2080501d0 con 0x7fe2180ab290 2024-03-29T10:50:15.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.659+0000 7fe1fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:15.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.659+0000 7fe1fffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:15.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.985+0000 7fe21dab5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:15.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.985+0000 7fe21dab5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:15.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.985+0000 7fe21dab5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:15.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.985+0000 7fe21dab5640 1 -- 172.21.15.98:0/348030057 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1f8005330 con 0x7fe2180ab290 2024-03-29T10:50:15.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.986+0000 7fe1fffff640 1 -- 172.21.15.98:0/348030057 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fe208052100 con 0x7fe2180ab290 2024-03-29T10:50:15.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.986+0000 7fe1fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:15.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.986+0000 7fe1fffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:15.970 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:50:15.970 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:15.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.990+0000 7fe21dab5640 10 librados: watch_flush enter 2024-03-29T10:50:15.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.991+0000 7fe21dab5640 10 librados: watch_flush exit 2024-03-29T10:50:15.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.991+0000 7fe21dab5640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T10:50:15.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.991+0000 7fe21dab5640 10 client.4448.objecter successfully canceled tick 2024-03-29T10:50:15.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.991+0000 7fe21dab5640 1 -- 172.21.15.98:0/348030057 >> v1:172.21.15.98:6816/1481799426 conn(0x7fe20406dde0 legacy=0x7fe2040702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:15.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.991+0000 7fe21dab5640 10 monclient: shutdown 2024-03-29T10:50:15.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.991+0000 7fe21dab5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:15.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.991+0000 7fe21dab5640 1 -- 172.21.15.98:0/348030057 >> v1:172.21.15.98:6790/0 conn(0x7fe2180ab290 legacy=0x7fe2181ffc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:15.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.991+0000 7fe21dab5640 1 -- 172.21.15.98:0/348030057 shutdown_connections 2024-03-29T10:50:15.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.991+0000 7fe21dab5640 1 -- 172.21.15.98:0/348030057 >> 172.21.15.98:0/348030057 conn(0x7fe218066a00 msgr2=0x7fe2180b6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:15.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.992+0000 7fe21dab5640 1 -- 172.21.15.98:0/348030057 shutdown_connections 2024-03-29T10:50:15.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.992+0000 7fe21dab5640 1 -- 172.21.15.98:0/348030057 wait complete. 2024-03-29T10:50:15.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:15.992+0000 7fe21dab5640 1 librados: shutdown 2024-03-29T10:50:15.993 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:22.799 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:22.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.964+0000 7f18745a3640 10 monclient: get_monmap_and_config 2024-03-29T10:50:22.946 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.964+0000 7f18745a3640 10 monclient: build_initial_monmap 2024-03-29T10:50:22.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.964+0000 7f18745a3640 10 monclient: monmap: 2024-03-29T10:50:22.947 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:22.947 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:22.947 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:50:22.965430+0000 2024-03-29T10:50:22.947 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:50:22.965430+0000 2024-03-29T10:50:22.947 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:22.947 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:22.947 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:22.947 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:50:22.947 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:50:22.947 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:22.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.969+0000 7f18745a3640 1 Processor -- start 2024-03-29T10:50:22.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.969+0000 7f18745a3640 1 -- start start 2024-03-29T10:50:22.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.969+0000 7f18745a3640 10 monclient: init 2024-03-29T10:50:22.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.970+0000 7f18745a3640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:22.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.970+0000 7f18745a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:22.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.970+0000 7f18745a3640 10 monclient(hunting): picked mon.noname-c con 0x7f186c0b0860 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:22.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.970+0000 7f18745a3640 10 monclient(hunting): picked mon.noname-b con 0x7f186c13ef30 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:22.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.973+0000 7f18745a3640 10 monclient(hunting): picked mon.noname-a con 0x7f186c0af8e0 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:22.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.973+0000 7f1872b40640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f186c0af8e0 0x7f186c149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:36813/0 (socket says 172.21.15.98:36813) 2024-03-29T10:50:22.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.973+0000 7f1872b40640 1 -- 172.21.15.98:0/2403101602 learned_addr learned my addr 172.21.15.98:0/2403101602 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:22.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.973+0000 7f18745a3640 1 -- 172.21.15.98:0/2403101602 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c0b0c40 con 0x7f186c0af8e0 2024-03-29T10:50:22.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.974+0000 7f18745a3640 1 -- 172.21.15.98:0/2403101602 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c0aed10 con 0x7f186c0b0860 2024-03-29T10:50:22.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.974+0000 7f18745a3640 1 -- 172.21.15.98:0/2403101602 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c0afd50 con 0x7f186c13ef30 2024-03-29T10:50:22.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.974+0000 7f18745a3640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:22.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.974+0000 7f187133d640 1 -- 172.21.15.98:0/2403101602 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070219576 0 0) 0x7f186c0b0c40 con 0x7f186c0af8e0 2024-03-29T10:50:22.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.974+0000 7f18745a3640 10 monclient(hunting): authenticate will time out at 1243.171337s 2024-03-29T10:50:22.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.974+0000 7f187133d640 10 monclient(hunting): _negotiate 2024-03-29T10:50:22.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.974+0000 7f187133d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:22.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.974+0000 7f187133d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:22.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.974+0000 7f187133d640 10 monclient(hunting): my global_id is 4422 2024-03-29T10:50:22.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.974+0000 7f187133d640 1 -- 172.21.15.98:0/2403101602 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1858003520 con 0x7f186c0af8e0 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.974+0000 7f187133d640 1 -- 172.21.15.98:0/2403101602 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3333048694 0 0) 0x7f1858003520 con 0x7f186c0af8e0 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f187133d640 1 -- 172.21.15.98:0/2403101602 >> v1:172.21.15.98:6790/0 conn(0x7f186c13ef30 legacy=0x7f186c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f187133d640 1 -- 172.21.15.98:0/2403101602 >> v1:172.21.15.189:6789/0 conn(0x7f186c0b0860 legacy=0x7f186c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f187133d640 10 monclient: _finish_hunting 0 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f187133d640 1 monclient: found mon.noname-a 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f187133d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f187133d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f187133d640 1 -- 172.21.15.98:0/2403101602 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c0ab1b0 con 0x7f186c0af8e0 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f187133d640 10 monclient: _finish_auth 0 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f187133d640 10 monclient: _check_auth_tickets 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f187133d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f187133d640 1 -- 172.21.15.98:0/2403101602 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f1864001a10 con 0x7f186c0af8e0 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f18745a3640 5 monclient: authenticate success, global_id 4422 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f18745a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:22.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f187133d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.975+0000 7f187133d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.976+0000 7f187133d640 10 monclient: dump: 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.976+0000 7f187133d640 1 -- 172.21.15.98:0/2403101602 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1864001c30 con 0x7f186c0af8e0 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.976+0000 7f18745a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:22.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.976+0000 7f187133d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.976+0000 7f187133d640 1 -- 172.21.15.98:0/2403101602 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f1864001fc0 con 0x7f186c0af8e0 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.976+0000 7f18745a3640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.976+0000 7f18745a3640 4 set_mon_vals no callback set 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.977+0000 7f18745a3640 10 monclient: shutdown 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.977+0000 7f187133d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.977+0000 7f187133d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.977+0000 7f187133d640 10 monclient: dump: 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:22.963 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:22.964 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:22.964 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:22.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.977+0000 7f18745a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:22.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.977+0000 7f18745a3640 1 -- 172.21.15.98:0/2403101602 >> v1:172.21.15.98:6789/0 conn(0x7f186c0af8e0 legacy=0x7f186c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:22.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.977+0000 7f18745a3640 1 -- 172.21.15.98:0/2403101602 shutdown_connections 2024-03-29T10:50:22.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.977+0000 7f18745a3640 1 -- 172.21.15.98:0/2403101602 >> 172.21.15.98:0/2403101602 conn(0x7f186c06b660 msgr2=0x7f186c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:22.965 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.979+0000 7f18745a3640 1 -- 172.21.15.98:0/2403101602 shutdown_connections 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.980+0000 7f18745a3640 1 -- 172.21.15.98:0/2403101602 wait complete. 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.981+0000 7f18745a3640 10 monclient: build_initial_monmap 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.981+0000 7f18745a3640 10 monclient: monmap: 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:22.966 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.981+0000 7f18745a3640 1 librados: starting msgr at 2024-03-29T10:50:22.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.981+0000 7f18745a3640 1 librados: starting objecter 2024-03-29T10:50:22.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.982+0000 7f18745a3640 1 Processor -- start 2024-03-29T10:50:22.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.982+0000 7f18745a3640 1 -- start start 2024-03-29T10:50:22.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.982+0000 7f18745a3640 1 librados: setting wanted keys 2024-03-29T10:50:22.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.982+0000 7f18745a3640 1 librados: calling monclient init 2024-03-29T10:50:22.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.982+0000 7f18745a3640 10 monclient: init 2024-03-29T10:50:22.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.983+0000 7f18745a3640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:22.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.983+0000 7f18745a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:22.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.983+0000 7f18745a3640 10 monclient(hunting): picked mon.noname-b con 0x7f186c0af8e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:22.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.983+0000 7f18745a3640 10 monclient(hunting): picked mon.noname-c con 0x7f186c0b0860 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:22.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1871b3e640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f186c0b0860 0x7f186c135830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:52204/0 (socket says 172.21.15.98:52204) 2024-03-29T10:50:22.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f18745a3640 10 monclient(hunting): picked mon.noname-a con 0x7f186c13ef30 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:22.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1872b40640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f186c13ef30 0x7f186c1d2f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37193/0 (socket says 172.21.15.98:37193) 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f18745a3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c135f40 con 0x7f186c13ef30 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f18745a3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c136270 con 0x7f186c0af8e0 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f18745a3640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186c1d3650 con 0x7f186c0b0860 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f18745a3640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f18745a3640 10 monclient(hunting): authenticate will time out at 1243.184296s 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1871b3e640 1 -- 172.21.15.98:0/3708252501 learned_addr learned my addr 172.21.15.98:0/3708252501 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 10 client.?.objecter ms_handle_connect 0x7f186c0b0860 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 10 client.?.objecter ms_handle_connect 0x7f186c13ef30 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120034796 0 0) 0x7f186c1d3650 con 0x7f186c0b0860 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:22.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:22.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:50:22.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1850004eb0 con 0x7f186c0b0860 2024-03-29T10:50:22.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013668386 0 0) 0x7f186c135f40 con 0x7f186c13ef30 2024-03-29T10:50:22.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:22.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:22.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:22.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 10 monclient(hunting): my global_id is 4425 2024-03-29T10:50:22.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.987+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f186c1d3650 con 0x7f186c13ef30 2024-03-29T10:50:22.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 416734376 0 0) 0x7f1850004eb0 con 0x7f186c0b0860 2024-03-29T10:50:22.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f186c135f40 con 0x7f186c0b0860 2024-03-29T10:50:22.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f185c001890 con 0x7f186c0b0860 2024-03-29T10:50:22.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:22.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 monclient(hunting): dump: 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3468750076 0 0) 0x7f186c1d3650 con 0x7f186c13ef30 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1850004eb0 con 0x7f186c13ef30 2024-03-29T10:50:22.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 client.?.objecter ms_handle_connect 0x7f186c0af8e0 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f1864001da0 con 0x7f186c13ef30 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 monclient(hunting): dump: 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:22.978 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2963790362 0 0) 0x7f186c135f40 con 0x7f186c0b0860 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 >> v1:172.21.15.189:6789/0 conn(0x7f186c0af8e0 legacy=0x7f186c138040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 >> v1:172.21.15.98:6789/0 conn(0x7f186c13ef30 legacy=0x7f186c1d2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 1 monclient: found mon.c 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c136790 con 0x7f186c0b0860 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 monclient: _finish_auth 0 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f18745a3640 5 monclient: authenticate success, global_id 4454 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f18745a3640 10 monclient: _renew_subs 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f18745a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:22.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f18745a3640 1 -- 172.21.15.98:0/3708252501 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186c136a40 con 0x7f186c0b0860 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f18745a3640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f18745a3640 10 monclient: _renew_subs 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f18745a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f18745a3640 1 -- 172.21.15.98:0/3708252501 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f186c0b3600 con 0x7f186c0b0860 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f18745a3640 1 librados: init done 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f185c003610 con 0x7f186c0b0860 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1862ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.988+0000 7f1870b3c640 4 set_mon_vals no callback set 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.989+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f185c001d80 con 0x7f186c0b0860 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.989+0000 7f1862ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.989+0000 7f1862ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.989+0000 7f1862ffd640 10 monclient: dump: 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:22.980 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:22.981 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:22.981 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:22.981 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:22.981 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:22.981 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:22.981 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:22.981 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:22.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.990+0000 7f18745a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:22.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.990+0000 7f18745a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:22.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.990+0000 7f18745a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:22.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.990+0000 7f18745a3640 1 -- 172.21.15.98:0/3708252501 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1854005080 con 0x7f186c0b0860 2024-03-29T10:50:22.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.992+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f185c003960 con 0x7f186c0b0860 2024-03-29T10:50:22.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.993+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f185c084650 con 0x7f186c0b0860 2024-03-29T10:50:22.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.993+0000 7f1862ffd640 10 client.4454.objecter ms_dispatch 0x7f186c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:22.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.993+0000 7f1862ffd640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:22.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.993+0000 7f1862ffd640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:22.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.993+0000 7f1862ffd640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T10:50:22.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.993+0000 7f187233f640 10 monclient: get_auth_request con 0x7f18500704a0 auth_method 0 2024-03-29T10:50:22.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:22.994+0000 7f1862ffd640 10 client.4454.objecter ms_handle_connect 0x7f18500704a0 2024-03-29T10:50:22.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.000+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f185c04ff90 con 0x7f186c0b0860 2024-03-29T10:50:22.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.000+0000 7f1862ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:22.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.000+0000 7f1862ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:23.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.323+0000 7f18745a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:23.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.323+0000 7f18745a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:23.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.323+0000 7f18745a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:23.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.323+0000 7f18745a3640 1 -- 172.21.15.98:0/3708252501 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1854005330 con 0x7f186c0b0860 2024-03-29T10:50:23.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.327+0000 7f1862ffd640 1 -- 172.21.15.98:0/3708252501 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f185c051ec0 con 0x7f186c0b0860 2024-03-29T10:50:23.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.327+0000 7f1862ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:23.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.327+0000 7f1862ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:23.309 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:50:23.309 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:23.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.331+0000 7f18745a3640 10 librados: watch_flush enter 2024-03-29T10:50:23.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.331+0000 7f18745a3640 10 librados: watch_flush exit 2024-03-29T10:50:23.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.331+0000 7f18745a3640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T10:50:23.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.331+0000 7f18745a3640 10 client.4454.objecter successfully canceled tick 2024-03-29T10:50:23.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.331+0000 7f18745a3640 1 -- 172.21.15.98:0/3708252501 >> v1:172.21.15.98:6816/1481799426 conn(0x7f18500704a0 legacy=0x7f1850072960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:23.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.331+0000 7f18745a3640 10 monclient: shutdown 2024-03-29T10:50:23.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.331+0000 7f18745a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:23.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.331+0000 7f18745a3640 1 -- 172.21.15.98:0/3708252501 >> v1:172.21.15.98:6790/0 conn(0x7f186c0b0860 legacy=0x7f186c135830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:23.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.331+0000 7f18745a3640 1 -- 172.21.15.98:0/3708252501 shutdown_connections 2024-03-29T10:50:23.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.331+0000 7f18745a3640 1 -- 172.21.15.98:0/3708252501 >> 172.21.15.98:0/3708252501 conn(0x7f186c06b660 msgr2=0x7f186c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:23.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.334+0000 7f18745a3640 1 -- 172.21.15.98:0/3708252501 shutdown_connections 2024-03-29T10:50:23.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.335+0000 7f18745a3640 1 -- 172.21.15.98:0/3708252501 wait complete. 2024-03-29T10:50:23.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:23.335+0000 7f18745a3640 1 librados: shutdown 2024-03-29T10:50:23.335 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:30.142 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:30.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.304+0000 7f12a3fff640 10 monclient: get_monmap_and_config 2024-03-29T10:50:30.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.304+0000 7f12a3fff640 10 monclient: build_initial_monmap 2024-03-29T10:50:30.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.304+0000 7f12a3fff640 10 monclient: monmap: 2024-03-29T10:50:30.287 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:30.287 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:30.287 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:50:30.304992+0000 2024-03-29T10:50:30.287 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:50:30.304992+0000 2024-03-29T10:50:30.287 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:30.287 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:30.287 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:30.287 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:50:30.287 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:50:30.287 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:30.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.304+0000 7f12a3fff640 1 Processor -- start 2024-03-29T10:50:30.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.305+0000 7f12a3fff640 1 -- start start 2024-03-29T10:50:30.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.305+0000 7f12a3fff640 10 monclient: init 2024-03-29T10:50:30.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.306+0000 7f12a3fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:30.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.306+0000 7f12a3fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:30.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.306+0000 7f12a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f12a414a490 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:30.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.306+0000 7f12a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f12a40aa520 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:30.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.306+0000 7f12a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f12a4149800 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:30.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.306+0000 7f12a3fff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a40aca50 con 0x7f12a4149800 2024-03-29T10:50:30.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.306+0000 7f12a3fff640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a40ac350 con 0x7f12a414a490 2024-03-29T10:50:30.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.306+0000 7f12a3fff640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a40ac630 con 0x7f12a40aa520 2024-03-29T10:50:30.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.306+0000 7f12a3fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:30.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.306+0000 7f12a3fff640 10 monclient(hunting): authenticate will time out at 1250.503163s 2024-03-29T10:50:30.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.309+0000 7f12a2ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f12a414a490 0x7f12a40aac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:36441/0 (socket says 172.21.15.98:36441) 2024-03-29T10:50:30.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.309+0000 7f12a2ffd640 1 -- 172.21.15.98:0/2273878684 learned_addr learned my addr 172.21.15.98:0/2273878684 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:30.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.310+0000 7f12a1ffb640 1 -- 172.21.15.98:0/2273878684 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846455013 0 0) 0x7f12a40ac350 con 0x7f12a414a490 2024-03-29T10:50:30.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.310+0000 7f12a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:30.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.310+0000 7f12a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:30.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.310+0000 7f12a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:30.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.310+0000 7f12a1ffb640 10 monclient(hunting): my global_id is 4429 2024-03-29T10:50:30.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.310+0000 7f12a1ffb640 1 -- 172.21.15.98:0/2273878684 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1288003520 con 0x7f12a414a490 2024-03-29T10:50:30.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.311+0000 7f12a1ffb640 1 -- 172.21.15.98:0/2273878684 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 681629357 0 0) 0x7f1288003520 con 0x7f12a414a490 2024-03-29T10:50:30.292 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.311+0000 7f12a1ffb640 1 -- 172.21.15.98:0/2273878684 >> v1:172.21.15.98:6790/0 conn(0x7f12a40aa520 legacy=0x7f12a40ad160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:30.293 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.312+0000 7f12a1ffb640 1 -- 172.21.15.98:0/2273878684 >> v1:172.21.15.98:6789/0 conn(0x7f12a4149800 legacy=0x7f12a4151ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:30.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:30.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a1ffb640 1 monclient: found mon.noname-c 2024-03-29T10:50:30.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:30.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:50:30.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a1ffb640 1 -- 172.21.15.98:0/2273878684 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a40aded0 con 0x7f12a414a490 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a3fff640 5 monclient: authenticate success, global_id 4429 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a1ffb640 1 -- 172.21.15.98:0/2273878684 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f129c001a10 con 0x7f12a414a490 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a1ffb640 10 monclient: dump: 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:30.298 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:30.299 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:30.299 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:30.299 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:30.299 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:30.299 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:30.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.315+0000 7f12a3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:30.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.316+0000 7f12a1ffb640 1 -- 172.21.15.98:0/2273878684 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f129c001ca0 con 0x7f12a414a490 2024-03-29T10:50:30.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.319+0000 7f12a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:30.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.319+0000 7f12a3fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:30.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.319+0000 7f12a3fff640 4 set_mon_vals no callback set 2024-03-29T10:50:30.301 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.319+0000 7f12a1ffb640 1 -- 172.21.15.98:0/2273878684 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f129c002030 con 0x7f12a414a490 2024-03-29T10:50:30.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.319+0000 7f12a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:30.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.319+0000 7f12a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:30.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.319+0000 7f12a1ffb640 10 monclient: dump: 2024-03-29T10:50:30.302 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:30.302 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:30.302 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:30.302 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:30.302 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:30.302 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:30.302 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:30.302 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:30.304 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:30.304 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:30.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.320+0000 7f12a3fff640 10 monclient: shutdown 2024-03-29T10:50:30.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.320+0000 7f12a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:30.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.320+0000 7f12a3fff640 1 -- 172.21.15.98:0/2273878684 >> v1:172.21.15.189:6789/0 conn(0x7f12a414a490 legacy=0x7f12a40aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:30.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.323+0000 7f12a3fff640 1 -- 172.21.15.98:0/2273878684 shutdown_connections 2024-03-29T10:50:30.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.323+0000 7f12a3fff640 1 -- 172.21.15.98:0/2273878684 >> 172.21.15.98:0/2273878684 conn(0x7f12a40b3a30 msgr2=0x7f12a40b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.323+0000 7f12a3fff640 1 -- 172.21.15.98:0/2273878684 shutdown_connections 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.323+0000 7f12a3fff640 1 -- 172.21.15.98:0/2273878684 wait complete. 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.324+0000 7f12a3fff640 10 monclient: build_initial_monmap 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.324+0000 7f12a3fff640 10 monclient: monmap: 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:50:30.307 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:30.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.324+0000 7f12a3fff640 1 librados: starting msgr at 2024-03-29T10:50:30.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.324+0000 7f12a3fff640 1 librados: starting objecter 2024-03-29T10:50:30.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.324+0000 7f12a3fff640 1 Processor -- start 2024-03-29T10:50:30.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.325+0000 7f12a3fff640 1 -- start start 2024-03-29T10:50:30.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.325+0000 7f12a3fff640 1 librados: setting wanted keys 2024-03-29T10:50:30.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.325+0000 7f12a3fff640 1 librados: calling monclient init 2024-03-29T10:50:30.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.325+0000 7f12a3fff640 10 monclient: init 2024-03-29T10:50:30.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.325+0000 7f12a3fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:30.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.325+0000 7f12a3fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:30.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.325+0000 7f12a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f12a40aa520 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:30.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.325+0000 7f12a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f12a4149800 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:30.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.326+0000 7f12a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f12a414a490 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:30.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.326+0000 7f12a3fff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a41db4f0 con 0x7f12a414a490 2024-03-29T10:50:30.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.326+0000 7f12a3fff640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a41dc3c0 con 0x7f12a40aa520 2024-03-29T10:50:30.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.326+0000 7f12a3fff640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a413f2b0 con 0x7f12a4149800 2024-03-29T10:50:30.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.326+0000 7f12a3fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:30.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.326+0000 7f12a3fff640 10 monclient(hunting): authenticate will time out at 1250.523019s 2024-03-29T10:50:30.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.329+0000 7f12a2ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f12a40aa520 0x7f12a4147f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:36541/0 (socket says 172.21.15.98:36541) 2024-03-29T10:50:30.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.329+0000 7f12a2ffd640 1 -- 172.21.15.98:0/311444193 learned_addr learned my addr 172.21.15.98:0/311444193 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:30.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.329+0000 7f129bfff640 10 client.?.objecter ms_handle_connect 0x7f12a40aa520 2024-03-29T10:50:30.312 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.329+0000 7f129bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:30.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.332+0000 7f129bfff640 10 client.?.objecter ms_handle_connect 0x7f12a414a490 2024-03-29T10:50:30.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.332+0000 7f129bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:30.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.332+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037567225 0 0) 0x7f12a41dc3c0 con 0x7f12a40aa520 2024-03-29T10:50:30.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.332+0000 7f129bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:30.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.332+0000 7f129bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:30.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.332+0000 7f129bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:30.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.332+0000 7f129bfff640 10 monclient(hunting): my global_id is 4432 2024-03-29T10:50:30.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.332+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f128c004eb0 con 0x7f12a40aa520 2024-03-29T10:50:30.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 client.?.objecter ms_handle_connect 0x7f12a4149800 2024-03-29T10:50:30.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:30.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500197342 0 0) 0x7f12a41db4f0 con 0x7f12a414a490 2024-03-29T10:50:30.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:30.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:30.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:30.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 monclient(hunting): my global_id is 4428 2024-03-29T10:50:30.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a41dc3c0 con 0x7f12a414a490 2024-03-29T10:50:30.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 334171406 0 0) 0x7f128c004eb0 con 0x7f12a40aa520 2024-03-29T10:50:30.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12a41db4f0 con 0x7f12a40aa520 2024-03-29T10:50:30.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125506037 0 0) 0x7f12a413f2b0 con 0x7f12a4149800 2024-03-29T10:50:30.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 monclient(hunting): my global_id is 4457 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f128c004eb0 con 0x7f12a4149800 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f129c001950 con 0x7f12a40aa520 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 monclient(hunting): dump: 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:30.322 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.334+0000 7f129bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3522636430 0 0) 0x7f12a41db4f0 con 0x7f12a40aa520 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 >> v1:172.21.15.98:6790/0 conn(0x7f12a4149800 legacy=0x7f12a41dbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 >> v1:172.21.15.98:6789/0 conn(0x7f12a414a490 legacy=0x7f12a41e5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f129bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f129bfff640 1 monclient: found mon.b 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f129bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f129bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a41db790 con 0x7f12a40aa520 2024-03-29T10:50:30.323 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f129bfff640 10 monclient: _finish_auth 0 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f129bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f129bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f12a3fff640 5 monclient: authenticate success, global_id 4432 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f12a3fff640 10 monclient: _renew_subs 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f12a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f12a3fff640 1 -- 172.21.15.98:0/311444193 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a4147170 con 0x7f12a40aa520 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f12a3fff640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f12a3fff640 10 monclient: _renew_subs 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f12a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f12a3fff640 1 -- 172.21.15.98:0/311444193 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a41447e0 con 0x7f12a40aa520 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.336+0000 7f12a3fff640 1 librados: init done 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.337+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f129c0015f0 con 0x7f12a40aa520 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.337+0000 7f129bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.337+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f129c004fb0 con 0x7f12a40aa520 2024-03-29T10:50:30.324 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.337+0000 7f129bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.337+0000 7f129bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.337+0000 7f129bfff640 10 monclient: dump: 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.337+0000 7f12a17fa640 4 set_mon_vals no callback set 2024-03-29T10:50:30.325 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.338+0000 7f12a3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:30.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.338+0000 7f12a3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:30.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.338+0000 7f12a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:30.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.338+0000 7f12a3fff640 1 -- 172.21.15.98:0/311444193 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1280005080 con 0x7f12a40aa520 2024-03-29T10:50:30.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.342+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f129c01af80 con 0x7f12a40aa520 2024-03-29T10:50:30.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.346+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f129c084dd0 con 0x7f12a40aa520 2024-03-29T10:50:30.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.346+0000 7f129bfff640 10 client.4432.objecter ms_dispatch 0x7f12a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:30.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.346+0000 7f129bfff640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:30.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.346+0000 7f129bfff640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:30.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.346+0000 7f129bfff640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-29T10:50:30.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.347+0000 7f12a27fc640 10 monclient: get_auth_request con 0x7f128c06dd60 auth_method 0 2024-03-29T10:50:30.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.347+0000 7f129bfff640 10 client.4432.objecter ms_handle_connect 0x7f128c06dd60 2024-03-29T10:50:30.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.350+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f129c050710 con 0x7f12a40aa520 2024-03-29T10:50:30.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.350+0000 7f129bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:30.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.350+0000 7f129bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:30.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.678+0000 7f12a3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:30.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.678+0000 7f12a3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:30.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.678+0000 7f12a3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:30.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.678+0000 7f12a3fff640 1 -- 172.21.15.98:0/311444193 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1280005330 con 0x7f12a40aa520 2024-03-29T10:50:30.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.679+0000 7f129bfff640 1 -- 172.21.15.98:0/311444193 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f129c052640 con 0x7f12a40aa520 2024-03-29T10:50:30.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.679+0000 7f129bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:30.661 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.679+0000 7f129bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:30.664 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:50:30.664 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.683+0000 7f12a3fff640 10 librados: watch_flush enter 2024-03-29T10:50:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.684+0000 7f12a3fff640 10 librados: watch_flush exit 2024-03-29T10:50:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.684+0000 7f12a3fff640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-29T10:50:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.684+0000 7f12a3fff640 10 client.4432.objecter successfully canceled tick 2024-03-29T10:50:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.684+0000 7f12a3fff640 1 -- 172.21.15.98:0/311444193 >> v1:172.21.15.98:6816/1481799426 conn(0x7f128c06dd60 legacy=0x7f128c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.684+0000 7f12a3fff640 10 monclient: shutdown 2024-03-29T10:50:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.684+0000 7f12a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.684+0000 7f12a3fff640 1 -- 172.21.15.98:0/311444193 >> v1:172.21.15.189:6789/0 conn(0x7f12a40aa520 legacy=0x7f12a4147f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.684+0000 7f12a3fff640 1 -- 172.21.15.98:0/311444193 shutdown_connections 2024-03-29T10:50:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.684+0000 7f12a3fff640 1 -- 172.21.15.98:0/311444193 >> 172.21.15.98:0/311444193 conn(0x7f12a40b3a30 msgr2=0x7f12a4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:30.666 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.684+0000 7f12a3fff640 1 -- 172.21.15.98:0/311444193 shutdown_connections 2024-03-29T10:50:30.667 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.684+0000 7f12a3fff640 1 -- 172.21.15.98:0/311444193 wait complete. 2024-03-29T10:50:30.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:30.687+0000 7f12a3fff640 1 librados: shutdown 2024-03-29T10:50:30.689 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:37.496 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:37.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.663+0000 7f9002c58640 10 monclient: get_monmap_and_config 2024-03-29T10:50:37.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.663+0000 7f9002c58640 10 monclient: build_initial_monmap 2024-03-29T10:50:37.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.663+0000 7f9002c58640 10 monclient: monmap: 2024-03-29T10:50:37.645 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:37.645 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:37.645 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:50:37.664392+0000 2024-03-29T10:50:37.645 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:50:37.664392+0000 2024-03-29T10:50:37.646 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:37.646 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:37.646 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:37.646 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:50:37.646 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:50:37.646 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:37.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.664+0000 7f9002c58640 1 Processor -- start 2024-03-29T10:50:37.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.664+0000 7f9002c58640 1 -- start start 2024-03-29T10:50:37.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.664+0000 7f9002c58640 10 monclient: init 2024-03-29T10:50:37.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.665+0000 7f9002c58640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:37.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.665+0000 7f9002c58640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:37.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.665+0000 7f9002c58640 10 monclient(hunting): picked mon.noname-a con 0x7f8ffc1457f0 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:37.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.665+0000 7f9002c58640 10 monclient(hunting): picked mon.noname-c con 0x7f8ffc13f650 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:37.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.665+0000 7f9002c58640 10 monclient(hunting): picked mon.noname-b con 0x7f8ffc0a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:37.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.665+0000 7f9002c58640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc0aaef0 con 0x7f8ffc1457f0 2024-03-29T10:50:37.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.665+0000 7f9002c58640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc145bd0 con 0x7f8ffc13f650 2024-03-29T10:50:37.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.665+0000 7f9002c58640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc145eb0 con 0x7f8ffc0a9d70 2024-03-29T10:50:37.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.665+0000 7f9002c58640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:37.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.665+0000 7f9002c58640 10 monclient(hunting): authenticate will time out at 1257.862600s 2024-03-29T10:50:37.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.668+0000 7f90011f5640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f8ffc0a9d70 0x7f8ffc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:60118/0 (socket says 172.21.15.98:60118) 2024-03-29T10:50:37.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f90011f5640 1 -- 172.21.15.98:0/3902812609 learned_addr learned my addr 172.21.15.98:0/3902812609 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:37.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 1 -- 172.21.15.98:0/3902812609 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490797370 0 0) 0x7f8ffc145bd0 con 0x7f8ffc13f650 2024-03-29T10:50:37.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:37.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:37.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:37.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 10 monclient(hunting): my global_id is 4435 2024-03-29T10:50:37.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 1 -- 172.21.15.98:0/3902812609 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe8003520 con 0x7f8ffc13f650 2024-03-29T10:50:37.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 1 -- 172.21.15.98:0/3902812609 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988767564 0 0) 0x7f8ffc145eb0 con 0x7f8ffc0a9d70 2024-03-29T10:50:37.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:37.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:37.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:37.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:50:37.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 1 -- 172.21.15.98:0/3902812609 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ffc145bd0 con 0x7f8ffc0a9d70 2024-03-29T10:50:37.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 1 -- 172.21.15.98:0/3902812609 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1491542135 0 0) 0x7f8fe8003520 con 0x7f8ffc13f650 2024-03-29T10:50:37.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 1 -- 172.21.15.98:0/3902812609 >> v1:172.21.15.98:6790/0 conn(0x7f8ffc0a9d70 legacy=0x7f8ffc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:37.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.669+0000 7f8ffb7fe640 1 -- 172.21.15.98:0/3902812609 >> v1:172.21.15.98:6789/0 conn(0x7f8ffc1457f0 legacy=0x7f8ffc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:37.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:50:37.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 1 monclient: found mon.noname-c 2024-03-29T10:50:37.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:37.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:50:37.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 1 -- 172.21.15.98:0/3902812609 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc146340 con 0x7f8ffc13f650 2024-03-29T10:50:37.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:50:37.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:50:37.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:37.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 1 -- 172.21.15.98:0/3902812609 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8fec001940 con 0x7f8ffc13f650 2024-03-29T10:50:37.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:37.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:37.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 10 monclient: dump: 2024-03-29T10:50:37.655 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:37.655 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:37.656 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:37.656 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:37.656 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:37.656 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:37.657 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:37.657 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:37.657 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:37.657 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:37.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f9002c58640 5 monclient: authenticate success, global_id 4435 2024-03-29T10:50:37.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f9002c58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:37.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 1 -- 172.21.15.98:0/3902812609 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fec001b90 con 0x7f8ffc13f650 2024-03-29T10:50:37.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:37.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f9002c58640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:37.657 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f9002c58640 4 set_mon_vals no callback set 2024-03-29T10:50:37.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 1 -- 172.21.15.98:0/3902812609 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8fec001f20 con 0x7f8ffc13f650 2024-03-29T10:50:37.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:37.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:37.658 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.670+0000 7f8ffb7fe640 10 monclient: dump: 2024-03-29T10:50:37.658 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:37.658 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:37.658 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:37.658 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:37.658 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:37.658 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:37.658 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:37.658 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:37.658 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:37.659 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:37.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.671+0000 7f9002c58640 10 monclient: shutdown 2024-03-29T10:50:37.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.671+0000 7f9002c58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:37.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.671+0000 7f9002c58640 1 -- 172.21.15.98:0/3902812609 >> v1:172.21.15.189:6789/0 conn(0x7f8ffc13f650 legacy=0x7f8ffc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:37.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.671+0000 7f9002c58640 1 -- 172.21.15.98:0/3902812609 shutdown_connections 2024-03-29T10:50:37.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.671+0000 7f9002c58640 1 -- 172.21.15.98:0/3902812609 >> 172.21.15.98:0/3902812609 conn(0x7f8ffc066a00 msgr2=0x7f8ffc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:37.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.671+0000 7f9002c58640 1 -- 172.21.15.98:0/3902812609 shutdown_connections 2024-03-29T10:50:37.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.671+0000 7f9002c58640 1 -- 172.21.15.98:0/3902812609 wait complete. 2024-03-29T10:50:37.668 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.672+0000 7f9002c58640 10 monclient: build_initial_monmap 2024-03-29T10:50:37.668 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.672+0000 7f9002c58640 10 monclient: monmap: 2024-03-29T10:50:37.668 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:37.668 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:37.668 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:50:37.668 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:50:37.668 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:37.668 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:37.668 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:37.669 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:50:37.669 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:50:37.669 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:37.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.672+0000 7f9002c58640 1 librados: starting msgr at 2024-03-29T10:50:37.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.672+0000 7f9002c58640 1 librados: starting objecter 2024-03-29T10:50:37.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.673+0000 7f9002c58640 1 Processor -- start 2024-03-29T10:50:37.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.673+0000 7f9002c58640 1 -- start start 2024-03-29T10:50:37.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.673+0000 7f9002c58640 1 librados: setting wanted keys 2024-03-29T10:50:37.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.673+0000 7f9002c58640 1 librados: calling monclient init 2024-03-29T10:50:37.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.673+0000 7f9002c58640 10 monclient: init 2024-03-29T10:50:37.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.674+0000 7f9002c58640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:37.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.674+0000 7f9002c58640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:37.669 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.674+0000 7f9002c58640 10 monclient(hunting): picked mon.noname-b con 0x7f8ffc0a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.674+0000 7f9002c58640 10 monclient(hunting): picked mon.noname-a con 0x7f8ffc1457f0 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.674+0000 7f9002c58640 10 monclient(hunting): picked mon.noname-c con 0x7f8ffc1df0f0 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.674+0000 7f9002c58640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc138dc0 con 0x7f8ffc1457f0 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.674+0000 7f9002c58640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc137f20 con 0x7f8ffc0a9d70 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.674+0000 7f9002c58640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc13a0d0 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.674+0000 7f9002c58640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.674+0000 7f9002c58640 10 monclient(hunting): authenticate will time out at 1257.871889s 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.677+0000 7f90011f5640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f8ffc1df0f0 0x7f8ffc137810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:35128/0 (socket says 172.21.15.98:35128) 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.677+0000 7f90011f5640 1 -- 172.21.15.98:0/4238855595 learned_addr learned my addr 172.21.15.98:0/4238855595 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.677+0000 7f8ff97fa640 10 client.?.objecter ms_handle_connect 0x7f8ffc1df0f0 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.677+0000 7f8ff97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393576510 0 0) 0x7f8ffc13a0d0 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:37.670 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 10 monclient(hunting): my global_id is 4463 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fdc004eb0 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 10 client.?.objecter ms_handle_connect 0x7f8ffc1457f0 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 347287003 0 0) 0x7f8fdc004eb0 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ffc13a0d0 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8fe4001930 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 10 monclient(hunting): dump: 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:37.671 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.678+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2902227697 0 0) 0x7f8ffc13a0d0 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 >> v1:172.21.15.189:6789/0 conn(0x7f8ffc0a9d70 legacy=0x7f8ffc1df6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 >> v1:172.21.15.98:6789/0 conn(0x7f8ffc1457f0 legacy=0x7f8ffc137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 1 monclient: found mon.c 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:37.672 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc141f60 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 10 monclient: _finish_auth 0 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fe40047e0 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8fe4004bb0 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ff97fa640 10 monclient: dump: 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:37.673 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f8ffaffd640 4 set_mon_vals no callback set 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.679+0000 7f9002c58640 5 monclient: authenticate success, global_id 4463 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.680+0000 7f9002c58640 10 monclient: _renew_subs 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.680+0000 7f9002c58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.680+0000 7f9002c58640 1 -- 172.21.15.98:0/4238855595 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ffc142a70 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.680+0000 7f9002c58640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.680+0000 7f9002c58640 10 monclient: _renew_subs 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.680+0000 7f9002c58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:37.674 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.680+0000 7f9002c58640 1 -- 172.21.15.98:0/4238855595 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ffc13a840 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.680+0000 7f9002c58640 1 librados: init done 2024-03-29T10:50:37.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.682+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f8fe4004d10 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.683+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f8fe4084ae0 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.683+0000 7f8ff97fa640 10 client.4463.objecter ms_dispatch 0x7f8ffc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:37.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.683+0000 7f8ff97fa640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:37.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.683+0000 7f8ff97fa640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:37.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.683+0000 7f90009f4640 10 monclient: get_auth_request con 0x7f8fdc06dca0 auth_method 0 2024-03-29T10:50:37.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.683+0000 7f9002c58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:37.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.683+0000 7f9002c58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:37.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.683+0000 7f9002c58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:37.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.683+0000 7f9002c58640 1 -- 172.21.15.98:0/4238855595 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ffc139380 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.684+0000 7f8ff97fa640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-29T10:50:37.675 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.684+0000 7f8ff97fa640 10 client.4463.objecter ms_handle_connect 0x7f8fdc06dca0 2024-03-29T10:50:37.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.693+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8fe40503c0 con 0x7f8ffc1df0f0 2024-03-29T10:50:37.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.693+0000 7f8ff97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:37.676 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:37.693+0000 7f8ff97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:38.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.022+0000 7f9002c58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:38.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.022+0000 7f9002c58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:38.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.022+0000 7f9002c58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:38.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.022+0000 7f9002c58640 1 -- 172.21.15.98:0/4238855595 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ffc075f10 con 0x7f8ffc1df0f0 2024-03-29T10:50:38.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.023+0000 7f8ff97fa640 1 -- 172.21.15.98:0/4238855595 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f8fe40522f0 con 0x7f8ffc1df0f0 2024-03-29T10:50:38.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.023+0000 7f8ff97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:38.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.023+0000 7f8ff97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:38.006 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:50:38.006 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:38.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.027+0000 7f9002c58640 10 librados: watch_flush enter 2024-03-29T10:50:38.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.028+0000 7f9002c58640 10 librados: watch_flush exit 2024-03-29T10:50:38.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.028+0000 7f9002c58640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-29T10:50:38.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.028+0000 7f9002c58640 10 client.4463.objecter successfully canceled tick 2024-03-29T10:50:38.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.028+0000 7f9002c58640 1 -- 172.21.15.98:0/4238855595 >> v1:172.21.15.98:6816/1481799426 conn(0x7f8fdc06dca0 legacy=0x7f8fdc070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:38.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.028+0000 7f9002c58640 10 monclient: shutdown 2024-03-29T10:50:38.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.028+0000 7f9002c58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:38.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.028+0000 7f9002c58640 1 -- 172.21.15.98:0/4238855595 >> v1:172.21.15.98:6790/0 conn(0x7f8ffc1df0f0 legacy=0x7f8ffc137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:38.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.028+0000 7f9002c58640 1 -- 172.21.15.98:0/4238855595 shutdown_connections 2024-03-29T10:50:38.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.028+0000 7f9002c58640 1 -- 172.21.15.98:0/4238855595 >> 172.21.15.98:0/4238855595 conn(0x7f8ffc066a00 msgr2=0x7f8ffc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:38.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.029+0000 7f9002c58640 1 -- 172.21.15.98:0/4238855595 shutdown_connections 2024-03-29T10:50:38.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.029+0000 7f9002c58640 1 -- 172.21.15.98:0/4238855595 wait complete. 2024-03-29T10:50:38.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:38.029+0000 7f9002c58640 1 librados: shutdown 2024-03-29T10:50:38.029 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:44.836 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:44.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.009+0000 7fb8ad4d3640 10 monclient: get_monmap_and_config 2024-03-29T10:50:44.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.009+0000 7fb8ad4d3640 10 monclient: build_initial_monmap 2024-03-29T10:50:44.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.009+0000 7fb8ad4d3640 10 monclient: monmap: 2024-03-29T10:50:44.993 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:44.993 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:44.993 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:50:45.010151+0000 2024-03-29T10:50:44.993 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:50:45.010151+0000 2024-03-29T10:50:44.993 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:44.993 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:44.993 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:44.993 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:50:44.994 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:50:44.994 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:44.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.010+0000 7fb8ad4d3640 1 Processor -- start 2024-03-29T10:50:44.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.010+0000 7fb8ad4d3640 1 -- start start 2024-03-29T10:50:44.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.010+0000 7fb8ad4d3640 10 monclient: init 2024-03-29T10:50:44.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.011+0000 7fb8ad4d3640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:44.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.011+0000 7fb8ad4d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:44.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.011+0000 7fb8ad4d3640 10 monclient(hunting): picked mon.noname-c con 0x7fb8a81457f0 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:44.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.011+0000 7fb8ad4d3640 10 monclient(hunting): picked mon.noname-b con 0x7fb8a813f650 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:44.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.011+0000 7fb8ad4d3640 10 monclient(hunting): picked mon.noname-a con 0x7fb8a80a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:44.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.011+0000 7fb8ad4d3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a80aaef0 con 0x7fb8a80a9d70 2024-03-29T10:50:44.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.011+0000 7fb8ad4d3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a8145bd0 con 0x7fb8a81457f0 2024-03-29T10:50:44.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.011+0000 7fb8ad4d3640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a8145eb0 con 0x7fb8a813f650 2024-03-29T10:50:44.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.011+0000 7fb8ad4d3640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:44.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.011+0000 7fb8ad4d3640 10 monclient(hunting): authenticate will time out at 1265.208156s 2024-03-29T10:50:44.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.014+0000 7fb8a77fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb8a80a9d70 0x7fb8a814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:33239/0 (socket says 172.21.15.98:33239) 2024-03-29T10:50:44.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.014+0000 7fb8a77fe640 1 -- 172.21.15.98:0/2317500927 learned_addr learned my addr 172.21.15.98:0/2317500927 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:45.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 1 -- 172.21.15.98:0/2317500927 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135426299 0 0) 0x7fb8a80aaef0 con 0x7fb8a80a9d70 2024-03-29T10:50:45.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:45.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:45.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:45.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient(hunting): my global_id is 4437 2024-03-29T10:50:45.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 1 -- 172.21.15.98:0/2317500927 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb890003520 con 0x7fb8a80a9d70 2024-03-29T10:50:45.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 1 -- 172.21.15.98:0/2317500927 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343137656 0 0) 0x7fb8a8145bd0 con 0x7fb8a81457f0 2024-03-29T10:50:45.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:45.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:45.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:45.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient(hunting): my global_id is 4438 2024-03-29T10:50:45.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 1 -- 172.21.15.98:0/2317500927 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a80aaef0 con 0x7fb8a81457f0 2024-03-29T10:50:45.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 1 -- 172.21.15.98:0/2317500927 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157916051 0 0) 0x7fb8a8145eb0 con 0x7fb8a813f650 2024-03-29T10:50:45.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:45.005 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:45.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:45.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:50:45.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 1 -- 172.21.15.98:0/2317500927 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a8145bd0 con 0x7fb8a813f650 2024-03-29T10:50:45.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 1 -- 172.21.15.98:0/2317500927 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3783440056 0 0) 0x7fb8a8145bd0 con 0x7fb8a813f650 2024-03-29T10:50:45.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 1 -- 172.21.15.98:0/2317500927 >> v1:172.21.15.189:6789/0 conn(0x7fb8a81457f0 legacy=0x7fb8a813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:45.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 1 -- 172.21.15.98:0/2317500927 >> v1:172.21.15.98:6789/0 conn(0x7fb8a80a9d70 legacy=0x7fb8a814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:45.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:45.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 1 monclient: found mon.noname-b 2024-03-29T10:50:45.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:45.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:50:45.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 1 -- 172.21.15.98:0/2317500927 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a8146340 con 0x7fb8a813f650 2024-03-29T10:50:45.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:45.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:45.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:45.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 1 -- 172.21.15.98:0/2317500927 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb89c001480 con 0x7fb8a813f650 2024-03-29T10:50:45.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:45.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:45.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.021+0000 7fb8a5ffb640 10 monclient: dump: 2024-03-29T10:50:45.007 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:45.007 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:45.007 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:45.007 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:45.007 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:45.007 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:45.008 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:45.008 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:45.008 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:45.008 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:45.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.022+0000 7fb8a5ffb640 1 -- 172.21.15.98:0/2317500927 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb89c0016a0 con 0x7fb8a813f650 2024-03-29T10:50:45.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.022+0000 7fb8a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:45.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.022+0000 7fb8a5ffb640 1 -- 172.21.15.98:0/2317500927 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb89c0019b0 con 0x7fb8a813f650 2024-03-29T10:50:45.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.022+0000 7fb8a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:45.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.022+0000 7fb8a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:45.009 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.022+0000 7fb8a5ffb640 10 monclient: dump: 2024-03-29T10:50:45.009 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:45.009 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:45.009 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:45.009 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:45.009 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:45.009 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:45.009 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:45.009 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:45.009 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:45.009 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:45.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.023+0000 7fb8ad4d3640 5 monclient: authenticate success, global_id 4466 2024-03-29T10:50:45.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.023+0000 7fb8ad4d3640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:45.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.023+0000 7fb8ad4d3640 4 set_mon_vals no callback set 2024-03-29T10:50:45.010 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.024+0000 7fb8ad4d3640 10 monclient: shutdown 2024-03-29T10:50:45.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.024+0000 7fb8ad4d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:45.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.024+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2317500927 >> v1:172.21.15.98:6790/0 conn(0x7fb8a813f650 legacy=0x7fb8a80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:45.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.028+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2317500927 shutdown_connections 2024-03-29T10:50:45.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.028+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2317500927 >> 172.21.15.98:0/2317500927 conn(0x7fb8a8066a00 msgr2=0x7fb8a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:45.011 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.029+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2317500927 shutdown_connections 2024-03-29T10:50:45.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.031+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2317500927 wait complete. 2024-03-29T10:50:45.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.032+0000 7fb8ad4d3640 10 monclient: build_initial_monmap 2024-03-29T10:50:45.016 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.032+0000 7fb8ad4d3640 10 monclient: monmap: 2024-03-29T10:50:45.016 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:45.016 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:45.016 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:50:45.016 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:50:45.017 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:45.017 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:45.017 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:45.017 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:50:45.017 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:50:45.017 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:45.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.032+0000 7fb8ad4d3640 1 librados: starting msgr at 2024-03-29T10:50:45.017 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.032+0000 7fb8ad4d3640 1 librados: starting objecter 2024-03-29T10:50:45.018 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.033+0000 7fb8ad4d3640 1 Processor -- start 2024-03-29T10:50:45.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.036+0000 7fb8ad4d3640 1 -- start start 2024-03-29T10:50:45.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.036+0000 7fb8ad4d3640 1 librados: setting wanted keys 2024-03-29T10:50:45.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.036+0000 7fb8ad4d3640 1 librados: calling monclient init 2024-03-29T10:50:45.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.036+0000 7fb8ad4d3640 10 monclient: init 2024-03-29T10:50:45.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.037+0000 7fb8ad4d3640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:45.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.037+0000 7fb8ad4d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:45.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.037+0000 7fb8ad4d3640 10 monclient(hunting): picked mon.noname-b con 0x7fb8a80a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:45.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.037+0000 7fb8ad4d3640 10 monclient(hunting): picked mon.noname-c con 0x7fb8a81457f0 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:45.021 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.037+0000 7fb8ad4d3640 10 monclient(hunting): picked mon.noname-a con 0x7fb8a8139920 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:45.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.037+0000 7fb8ad4d3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a8141770 con 0x7fb8a8139920 2024-03-29T10:50:45.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.037+0000 7fb8ad4d3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a8142980 con 0x7fb8a80a9d70 2024-03-29T10:50:45.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.037+0000 7fb8ad4d3640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a81425f0 con 0x7fb8a81457f0 2024-03-29T10:50:45.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.037+0000 7fb8ad4d3640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:45.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.037+0000 7fb8ad4d3640 10 monclient(hunting): authenticate will time out at 1265.234162s 2024-03-29T10:50:45.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.041+0000 7fb8a6ffd640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb8a80a9d70 0x7fb8a8141ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:50393/0 (socket says 172.21.15.98:50393) 2024-03-29T10:50:45.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.041+0000 7fb8a6ffd640 1 -- 172.21.15.98:0/2934122398 learned_addr learned my addr 172.21.15.98:0/2934122398 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:45.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.041+0000 7fb88f7fe640 10 client.?.objecter ms_handle_connect 0x7fb8a80a9d70 2024-03-29T10:50:45.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.042+0000 7fb88f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:45.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.043+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950601108 0 0) 0x7fb8a8142980 con 0x7fb8a80a9d70 2024-03-29T10:50:45.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.043+0000 7fb88f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:45.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.043+0000 7fb88f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:45.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.043+0000 7fb88f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:45.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.043+0000 7fb88f7fe640 10 monclient(hunting): my global_id is 4441 2024-03-29T10:50:45.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.043+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb894004eb0 con 0x7fb8a80a9d70 2024-03-29T10:50:45.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.043+0000 7fb88f7fe640 10 client.?.objecter ms_handle_connect 0x7fb8a81457f0 2024-03-29T10:50:45.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.043+0000 7fb88f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:45.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.043+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3399435245 0 0) 0x7fb894004eb0 con 0x7fb8a80a9d70 2024-03-29T10:50:45.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8a8142980 con 0x7fb8a80a9d70 2024-03-29T10:50:45.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb8980019a0 con 0x7fb8a80a9d70 2024-03-29T10:50:45.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 10 monclient(hunting): dump: 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146357256 0 0) 0x7fb8a81425f0 con 0x7fb8a81457f0 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:45.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb894004eb0 con 0x7fb8a81457f0 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1973672684 0 0) 0x7fb8a8142980 con 0x7fb8a80a9d70 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 >> v1:172.21.15.98:6790/0 conn(0x7fb8a81457f0 legacy=0x7fb8a81d6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 >> v1:172.21.15.98:6789/0 conn(0x7fb8a8139920 legacy=0x7fb8a81d6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 1 monclient: found mon.b 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a81e1910 con 0x7fb8a80a9d70 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:50:45.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.044+0000 7fb88f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:45.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.048+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb898001b00 con 0x7fb8a80a9d70 2024-03-29T10:50:45.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.048+0000 7fb88f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:45.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.048+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb898003a60 con 0x7fb8a80a9d70 2024-03-29T10:50:45.029 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.048+0000 7fb88f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:45.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.048+0000 7fb88f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:45.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.048+0000 7fb88f7fe640 10 monclient: dump: 2024-03-29T10:50:45.030 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:45.030 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:45.030 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:45.030 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:45.030 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:45.030 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:45.030 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:45.030 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:45.030 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:45.030 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:45.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.048+0000 7fb8a57fa640 4 set_mon_vals no callback set 2024-03-29T10:50:45.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.049+0000 7fb8ad4d3640 5 monclient: authenticate success, global_id 4441 2024-03-29T10:50:45.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.049+0000 7fb8ad4d3640 10 monclient: _renew_subs 2024-03-29T10:50:45.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.049+0000 7fb8ad4d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:45.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.049+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2934122398 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a81de8e0 con 0x7fb8a80a9d70 2024-03-29T10:50:45.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.049+0000 7fb8ad4d3640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:45.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.049+0000 7fb8ad4d3640 10 monclient: _renew_subs 2024-03-29T10:50:45.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.049+0000 7fb8ad4d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:45.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.049+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2934122398 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a81dedc0 con 0x7fb8a80a9d70 2024-03-29T10:50:45.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.049+0000 7fb8ad4d3640 1 librados: init done 2024-03-29T10:50:45.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.052+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fb898003bc0 con 0x7fb8a80a9d70 2024-03-29T10:50:45.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.052+0000 7fb8ad4d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:45.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.053+0000 7fb8ad4d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:45.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.053+0000 7fb8ad4d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:45.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.053+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2934122398 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a8142780 con 0x7fb8a80a9d70 2024-03-29T10:50:45.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.054+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fb898084b10 con 0x7fb8a80a9d70 2024-03-29T10:50:45.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.054+0000 7fb88f7fe640 10 client.4441.objecter ms_dispatch 0x7fb8a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:45.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.054+0000 7fb88f7fe640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:45.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.054+0000 7fb88f7fe640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:45.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.054+0000 7fb88f7fe640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-29T10:50:45.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.055+0000 7fb8a67fc640 10 monclient: get_auth_request con 0x7fb89406dd60 auth_method 0 2024-03-29T10:50:45.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.056+0000 7fb88f7fe640 10 client.4441.objecter ms_handle_connect 0x7fb89406dd60 2024-03-29T10:50:45.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.063+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8980503f0 con 0x7fb8a80a9d70 2024-03-29T10:50:45.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.063+0000 7fb88f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:45.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.063+0000 7fb88f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:45.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.382+0000 7fb8ad4d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:45.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.382+0000 7fb8ad4d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:45.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.382+0000 7fb8ad4d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:45.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.382+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2934122398 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8a8075f10 con 0x7fb8a80a9d70 2024-03-29T10:50:45.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.386+0000 7fb88f7fe640 1 -- 172.21.15.98:0/2934122398 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fb898052320 con 0x7fb8a80a9d70 2024-03-29T10:50:45.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.386+0000 7fb88f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:45.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.386+0000 7fb88f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:45.368 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:50:45.368 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:45.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.389+0000 7fb8ad4d3640 10 librados: watch_flush enter 2024-03-29T10:50:45.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.390+0000 7fb8ad4d3640 10 librados: watch_flush exit 2024-03-29T10:50:45.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.390+0000 7fb8ad4d3640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-29T10:50:45.371 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.390+0000 7fb8ad4d3640 10 client.4441.objecter successfully canceled tick 2024-03-29T10:50:45.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.390+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2934122398 >> v1:172.21.15.98:6816/1481799426 conn(0x7fb89406dd60 legacy=0x7fb894070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:45.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.390+0000 7fb8ad4d3640 10 monclient: shutdown 2024-03-29T10:50:45.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.390+0000 7fb8ad4d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:45.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.390+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2934122398 >> v1:172.21.15.189:6789/0 conn(0x7fb8a80a9d70 legacy=0x7fb8a8141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:45.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.390+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2934122398 shutdown_connections 2024-03-29T10:50:45.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.390+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2934122398 >> 172.21.15.98:0/2934122398 conn(0x7fb8a8066a00 msgr2=0x7fb8a813b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:45.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.391+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2934122398 shutdown_connections 2024-03-29T10:50:45.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.391+0000 7fb8ad4d3640 1 -- 172.21.15.98:0/2934122398 wait complete. 2024-03-29T10:50:45.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:45.391+0000 7fb8ad4d3640 1 librados: shutdown 2024-03-29T10:50:45.392 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:52.200 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:52.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.370+0000 7fd529918640 10 monclient: get_monmap_and_config 2024-03-29T10:50:52.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.370+0000 7fd529918640 10 monclient: build_initial_monmap 2024-03-29T10:50:52.352 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.370+0000 7fd529918640 10 monclient: monmap: 2024-03-29T10:50:52.352 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:52.352 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:52.352 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:50:52.371259+0000 2024-03-29T10:50:52.352 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:50:52.371259+0000 2024-03-29T10:50:52.352 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:52.352 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:52.352 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:52.352 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:50:52.352 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:50:52.353 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:52.353 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.371+0000 7fd529918640 1 Processor -- start 2024-03-29T10:50:52.353 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.371+0000 7fd529918640 1 -- start start 2024-03-29T10:50:52.353 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.371+0000 7fd529918640 10 monclient: init 2024-03-29T10:50:52.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.372+0000 7fd529918640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:52.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.372+0000 7fd529918640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:52.359 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.372+0000 7fd529918640 10 monclient(hunting): picked mon.noname-b con 0x7fd51c006d80 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.372+0000 7fd528916640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fd51c006d80 0x7fd51c0c46e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:57946/0 (socket says 172.21.15.98:57946) 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.372+0000 7fd528916640 1 -- 172.21.15.98:0/90576013 learned_addr learned my addr 172.21.15.98:0/90576013 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.372+0000 7fd529918640 10 monclient(hunting): picked mon.noname-a con 0x7fd51c0c4df0 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.372+0000 7fd529918640 10 monclient(hunting): picked mon.noname-c con 0x7fd51c0cb5e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.372+0000 7fd529918640 1 -- 172.21.15.98:0/90576013 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c008d80 con 0x7fd51c0c4df0 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.372+0000 7fd529918640 1 -- 172.21.15.98:0/90576013 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c007160 con 0x7fd51c0cb5e0 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.372+0000 7fd529918640 1 -- 172.21.15.98:0/90576013 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c0302b0 con 0x7fd51c006d80 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.372+0000 7fd529918640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.372+0000 7fd529918640 10 monclient(hunting): authenticate will time out at 1272.569922s 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 1 -- 172.21.15.98:0/90576013 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485913725 0 0) 0x7fd51c0302b0 con 0x7fd51c006d80 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:52.360 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 1 -- 172.21.15.98:0/90576013 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd50c003520 con 0x7fd51c006d80 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 1 -- 172.21.15.98:0/90576013 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312440145 0 0) 0x7fd51c007160 con 0x7fd51c0cb5e0 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 10 monclient(hunting): my global_id is 4444 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 1 -- 172.21.15.98:0/90576013 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd51c0302b0 con 0x7fd51c0cb5e0 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 1 -- 172.21.15.98:0/90576013 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 441991770 0 0) 0x7fd50c003520 con 0x7fd51c006d80 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 1 -- 172.21.15.98:0/90576013 >> v1:172.21.15.189:6789/0 conn(0x7fd51c0cb5e0 legacy=0x7fd51c0d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.373+0000 7fd5237fe640 1 -- 172.21.15.98:0/90576013 >> v1:172.21.15.98:6789/0 conn(0x7fd51c0c4df0 legacy=0x7fd51c0caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 10 monclient: _finish_hunting 0 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 1 monclient: found mon.noname-b 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:52.361 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 1 -- 172.21.15.98:0/90576013 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c030520 con 0x7fd51c006d80 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 10 monclient: _finish_auth 0 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 10 monclient: _check_auth_tickets 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 1 -- 172.21.15.98:0/90576013 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd514001690 con 0x7fd51c006d80 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd529918640 5 monclient: authenticate success, global_id 4472 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd529918640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 10 monclient: dump: 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:52.362 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd529918640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 1 -- 172.21.15.98:0/90576013 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd514001920 con 0x7fd51c006d80 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd5237fe640 1 -- 172.21.15.98:0/90576013 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd514001cb0 con 0x7fd51c006d80 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd529918640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.374+0000 7fd529918640 4 set_mon_vals no callback set 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.375+0000 7fd5237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:52.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.375+0000 7fd5237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:52.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.375+0000 7fd5237fe640 10 monclient: dump: 2024-03-29T10:50:52.364 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:52.364 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:52.364 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:52.364 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.375+0000 7fd529918640 10 monclient: shutdown 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.375+0000 7fd529918640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.375+0000 7fd529918640 1 -- 172.21.15.98:0/90576013 >> v1:172.21.15.98:6790/0 conn(0x7fd51c006d80 legacy=0x7fd51c0c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.375+0000 7fd529918640 1 -- 172.21.15.98:0/90576013 shutdown_connections 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.375+0000 7fd529918640 1 -- 172.21.15.98:0/90576013 >> 172.21.15.98:0/90576013 conn(0x7fd51c005c00 msgr2=0x7fd51c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.376+0000 7fd529918640 1 -- 172.21.15.98:0/90576013 shutdown_connections 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.376+0000 7fd529918640 1 -- 172.21.15.98:0/90576013 wait complete. 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.377+0000 7fd529918640 10 monclient: build_initial_monmap 2024-03-29T10:50:52.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.377+0000 7fd529918640 10 monclient: monmap: 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.377+0000 7fd529918640 1 librados: starting msgr at 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.377+0000 7fd529918640 1 librados: starting objecter 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.377+0000 7fd529918640 1 Processor -- start 2024-03-29T10:50:52.366 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.378+0000 7fd529918640 1 -- start start 2024-03-29T10:50:52.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.378+0000 7fd529918640 1 librados: setting wanted keys 2024-03-29T10:50:52.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.378+0000 7fd529918640 1 librados: calling monclient init 2024-03-29T10:50:52.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.378+0000 7fd529918640 10 monclient: init 2024-03-29T10:50:52.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.379+0000 7fd529918640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:52.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.379+0000 7fd529918640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:52.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.379+0000 7fd529918640 10 monclient(hunting): picked mon.noname-a con 0x7fd51c006d80 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:52.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.379+0000 7fd528916640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd51c006d80 0x7fd51c0bb530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:35346/0 (socket says 172.21.15.98:35346) 2024-03-29T10:50:52.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.379+0000 7fd528916640 1 -- 172.21.15.98:0/1472225932 learned_addr learned my addr 172.21.15.98:0/1472225932 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:52.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.379+0000 7fd529918640 10 monclient(hunting): picked mon.noname-b con 0x7fd51c0c4df0 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:52.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.379+0000 7fd529918640 10 monclient(hunting): picked mon.noname-c con 0x7fd51c0cb5e0 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.379+0000 7fd5217fa640 10 client.?.objecter ms_handle_connect 0x7fd51c006d80 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.379+0000 7fd5217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 10 client.?.objecter ms_handle_connect 0x7fd51c0cb5e0 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.379+0000 7fd529918640 1 -- 172.21.15.98:0/1472225932 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c0c7f20 con 0x7fd51c006d80 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd529918640 1 -- 172.21.15.98:0/1472225932 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c0c68d0 con 0x7fd51c0c4df0 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd529918640 1 -- 172.21.15.98:0/1472225932 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c034f30 con 0x7fd51c0cb5e0 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd529918640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd529918640 10 monclient(hunting): authenticate will time out at 1272.577143s 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 10 client.?.objecter ms_handle_connect 0x7fd51c0c4df0 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602522925 0 0) 0x7fd51c034f30 con 0x7fd51c0cb5e0 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:52.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd510004eb0 con 0x7fd51c0cb5e0 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897437630 0 0) 0x7fd51c0c68d0 con 0x7fd51c0c4df0 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 10 monclient(hunting): my global_id is 4447 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.380+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd51c034f30 con 0x7fd51c0c4df0 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1418417326 0 0) 0x7fd51c034f30 con 0x7fd51c0c4df0 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd51c0c68d0 con 0x7fd51c0c4df0 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd518001860 con 0x7fd51c0c4df0 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:52.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 10 monclient(hunting): dump: 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825423438 0 0) 0x7fd51c0c7f20 con 0x7fd51c006d80 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:52.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 10 monclient(hunting): my global_id is 4440 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd51c034f30 con 0x7fd51c006d80 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3036212508 0 0) 0x7fd51c0c68d0 con 0x7fd51c0c4df0 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 >> v1:172.21.15.98:6790/0 conn(0x7fd51c0cb5e0 legacy=0x7fd51c1681d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 >> v1:172.21.15.98:6789/0 conn(0x7fd51c006d80 legacy=0x7fd51c0bb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 10 monclient: _finish_hunting 0 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 1 monclient: found mon.b 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c0bbc40 con 0x7fd51c0c4df0 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 10 monclient: _finish_auth 0 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 10 monclient: _check_auth_tickets 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd5217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd529918640 5 monclient: authenticate success, global_id 4447 2024-03-29T10:50:52.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd529918640 10 monclient: _renew_subs 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd529918640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd529918640 1 -- 172.21.15.98:0/1472225932 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51c0bbef0 con 0x7fd51c0c4df0 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd529918640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd529918640 10 monclient: _renew_subs 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd529918640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd529918640 1 -- 172.21.15.98:0/1472225932 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51c039b00 con 0x7fd51c0c4df0 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.381+0000 7fd529918640 1 librados: init done 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.382+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5180019c0 con 0x7fd51c0c4df0 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.382+0000 7fd5217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.382+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd518003870 con 0x7fd51c0c4df0 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.382+0000 7fd5217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.382+0000 7fd5217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.382+0000 7fd5217fa640 10 monclient: dump: 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:52.377 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.382+0000 7fd5227fc640 4 set_mon_vals no callback set 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.383+0000 7fd529918640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.383+0000 7fd529918640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.383+0000 7fd529918640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.383+0000 7fd529918640 1 -- 172.21.15.98:0/1472225932 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd504005080 con 0x7fd51c0c4df0 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.385+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fd5180039d0 con 0x7fd51c0c4df0 2024-03-29T10:50:52.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.385+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fd5180837f0 con 0x7fd51c0c4df0 2024-03-29T10:50:52.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.385+0000 7fd5217fa640 10 client.4447.objecter ms_dispatch 0x7fd524000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:52.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.385+0000 7fd5217fa640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:52.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.385+0000 7fd5217fa640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:52.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.385+0000 7fd528916640 10 monclient: get_auth_request con 0x7fd51006dd30 auth_method 0 2024-03-29T10:50:52.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.385+0000 7fd5217fa640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T10:50:52.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.385+0000 7fd5217fa640 10 client.4447.objecter ms_handle_connect 0x7fd51006dd30 2024-03-29T10:50:52.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.393+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd51804f0d0 con 0x7fd51c0c4df0 2024-03-29T10:50:52.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.393+0000 7fd5217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:52.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.393+0000 7fd5217fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.732+0000 7fd529918640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.732+0000 7fd529918640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.732+0000 7fd529918640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:50:52.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.732+0000 7fd529918640 1 -- 172.21.15.98:0/1472225932 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd504005330 con 0x7fd51c0c4df0 2024-03-29T10:50:52.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.734+0000 7fd5217fa640 1 -- 172.21.15.98:0/1472225932 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fd518051000 con 0x7fd51c0c4df0 2024-03-29T10:50:52.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.734+0000 7fd5217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:52.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.734+0000 7fd5217fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:52.716 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:50:52.716 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:52.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.738+0000 7fd529918640 10 librados: watch_flush enter 2024-03-29T10:50:52.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.738+0000 7fd529918640 10 librados: watch_flush exit 2024-03-29T10:50:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.738+0000 7fd529918640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T10:50:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.738+0000 7fd529918640 10 client.4447.objecter successfully canceled tick 2024-03-29T10:50:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.738+0000 7fd529918640 1 -- 172.21.15.98:0/1472225932 >> v1:172.21.15.98:6816/1481799426 conn(0x7fd51006dd30 legacy=0x7fd5100701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.738+0000 7fd529918640 10 monclient: shutdown 2024-03-29T10:50:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.738+0000 7fd529918640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.738+0000 7fd529918640 1 -- 172.21.15.98:0/1472225932 >> v1:172.21.15.189:6789/0 conn(0x7fd51c0c4df0 legacy=0x7fd51c15c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.738+0000 7fd529918640 1 -- 172.21.15.98:0/1472225932 shutdown_connections 2024-03-29T10:50:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.738+0000 7fd529918640 1 -- 172.21.15.98:0/1472225932 >> 172.21.15.98:0/1472225932 conn(0x7fd51c005c00 msgr2=0x7fd51c0c3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.738+0000 7fd529918640 1 -- 172.21.15.98:0/1472225932 shutdown_connections 2024-03-29T10:50:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.738+0000 7fd529918640 1 -- 172.21.15.98:0/1472225932 wait complete. 2024-03-29T10:50:52.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:52.738+0000 7fd529918640 1 librados: shutdown 2024-03-29T10:50:52.736 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:59.542 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:59.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.720+0000 7fc70d443640 10 monclient: get_monmap_and_config 2024-03-29T10:50:59.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.720+0000 7fc70d443640 10 monclient: build_initial_monmap 2024-03-29T10:50:59.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.720+0000 7fc70d443640 10 monclient: monmap: 2024-03-29T10:50:59.702 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:59.702 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:59.702 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:50:59.720917+0000 2024-03-29T10:50:59.702 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:50:59.720917+0000 2024-03-29T10:50:59.702 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:59.702 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:59.702 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:59.702 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:50:59.702 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:50:59.702 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:59.704 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.722+0000 7fc70d443640 1 Processor -- start 2024-03-29T10:50:59.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.723+0000 7fc70d443640 1 -- start start 2024-03-29T10:50:59.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.723+0000 7fc70d443640 10 monclient: init 2024-03-29T10:50:59.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.723+0000 7fc70d443640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:59.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.723+0000 7fc70d443640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:59.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.723+0000 7fc70d443640 10 monclient(hunting): picked mon.noname-a con 0x7fc7080b0770 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:59.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.723+0000 7fc70d443640 10 monclient(hunting): picked mon.noname-c con 0x7fc70814a700 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:59.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.723+0000 7fc70d443640 10 monclient(hunting): picked mon.noname-b con 0x7fc7080ac5a0 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:59.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.723+0000 7fc70d443640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7080aba70 con 0x7fc7080b0770 2024-03-29T10:50:59.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.724+0000 7fc70d443640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7080aecb0 con 0x7fc70814a700 2024-03-29T10:50:59.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.724+0000 7fc70d443640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7080af150 con 0x7fc7080ac5a0 2024-03-29T10:50:59.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.724+0000 7fc70d443640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:59.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.724+0000 7fc70d443640 10 monclient(hunting): authenticate will time out at 1279.921007s 2024-03-29T10:50:59.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.730+0000 7fc7077fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fc7080ac5a0 0x7fc708151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:59873/0 (socket says 172.21.15.98:59873) 2024-03-29T10:50:59.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.730+0000 7fc7077fe640 1 -- 172.21.15.98:0/3352849793 learned_addr learned my addr 172.21.15.98:0/3352849793 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:59.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.730+0000 7fc705ffb640 1 -- 172.21.15.98:0/3352849793 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166271740 0 0) 0x7fc7080af150 con 0x7fc7080ac5a0 2024-03-29T10:50:59.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.730+0000 7fc705ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:59.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.730+0000 7fc705ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:59.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.730+0000 7fc705ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:59.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.730+0000 7fc705ffb640 10 monclient(hunting): my global_id is 4478 2024-03-29T10:50:59.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.730+0000 7fc705ffb640 1 -- 172.21.15.98:0/3352849793 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f0003520 con 0x7fc7080ac5a0 2024-03-29T10:50:59.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 1 -- 172.21.15.98:0/3352849793 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953474157 0 0) 0x7fc7080aecb0 con 0x7fc70814a700 2024-03-29T10:50:59.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:59.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:59.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:59.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient(hunting): my global_id is 4450 2024-03-29T10:50:59.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 1 -- 172.21.15.98:0/3352849793 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7080af150 con 0x7fc70814a700 2024-03-29T10:50:59.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 1 -- 172.21.15.98:0/3352849793 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588770043 0 0) 0x7fc7080aba70 con 0x7fc7080b0770 2024-03-29T10:50:59.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:50:59.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:59.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:59.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient(hunting): my global_id is 4443 2024-03-29T10:50:59.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 1 -- 172.21.15.98:0/3352849793 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7080aecb0 con 0x7fc7080b0770 2024-03-29T10:50:59.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 1 -- 172.21.15.98:0/3352849793 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3478189344 0 0) 0x7fc6f0003520 con 0x7fc7080ac5a0 2024-03-29T10:50:59.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 1 -- 172.21.15.98:0/3352849793 >> v1:172.21.15.189:6789/0 conn(0x7fc70814a700 legacy=0x7fc7080abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:59.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 1 -- 172.21.15.98:0/3352849793 >> v1:172.21.15.98:6789/0 conn(0x7fc7080b0770 legacy=0x7fc708149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:59.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:50:59.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 1 monclient: found mon.noname-b 2024-03-29T10:50:59.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:59.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:50:59.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 1 -- 172.21.15.98:0/3352849793 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7080ac9d0 con 0x7fc7080ac5a0 2024-03-29T10:50:59.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient: _finish_auth 0 2024-03-29T10:50:59.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:50:59.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:59.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 1 -- 172.21.15.98:0/3352849793 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fc6fc001690 con 0x7fc7080ac5a0 2024-03-29T10:50:59.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc70d443640 5 monclient: authenticate success, global_id 4478 2024-03-29T10:50:59.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc70d443640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:59.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:59.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:59.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc705ffb640 10 monclient: dump: 2024-03-29T10:50:59.716 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:59.716 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:59.717 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:59.717 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:59.717 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:59.717 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:59.717 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:59.717 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:59.717 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:59.717 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:59.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.731+0000 7fc70d443640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:59.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.735+0000 7fc705ffb640 1 -- 172.21.15.98:0/3352849793 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6fc0017f0 con 0x7fc7080ac5a0 2024-03-29T10:50:59.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.735+0000 7fc705ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:59.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.735+0000 7fc705ffb640 1 -- 172.21.15.98:0/3352849793 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fc6fc001c00 con 0x7fc7080ac5a0 2024-03-29T10:50:59.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.735+0000 7fc705ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:59.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.735+0000 7fc705ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:59.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.735+0000 7fc705ffb640 10 monclient: dump: 2024-03-29T10:50:59.718 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:59.718 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:59.718 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:59.718 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:59.718 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:59.719 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:59.719 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:59.719 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:59.719 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:59.719 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:59.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.736+0000 7fc70d443640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:59.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.736+0000 7fc70d443640 4 set_mon_vals no callback set 2024-03-29T10:50:59.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.737+0000 7fc70d443640 10 monclient: shutdown 2024-03-29T10:50:59.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.737+0000 7fc70d443640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:59.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.737+0000 7fc70d443640 1 -- 172.21.15.98:0/3352849793 >> v1:172.21.15.98:6790/0 conn(0x7fc7080ac5a0 legacy=0x7fc708151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:59.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.737+0000 7fc70d443640 1 -- 172.21.15.98:0/3352849793 shutdown_connections 2024-03-29T10:50:59.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.737+0000 7fc70d443640 1 -- 172.21.15.98:0/3352849793 >> 172.21.15.98:0/3352849793 conn(0x7fc70806b660 msgr2=0x7fc70806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:59.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.738+0000 7fc70d443640 1 -- 172.21.15.98:0/3352849793 shutdown_connections 2024-03-29T10:50:59.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.741+0000 7fc70d443640 1 -- 172.21.15.98:0/3352849793 wait complete. 2024-03-29T10:50:59.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.742+0000 7fc70d443640 10 monclient: build_initial_monmap 2024-03-29T10:50:59.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.742+0000 7fc70d443640 10 monclient: monmap: 2024-03-29T10:50:59.724 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:50:59.724 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:59.724 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:50:59.724 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:50:59.724 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:59.724 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:59.725 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:50:59.725 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:50:59.725 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:50:59.725 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:59.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.743+0000 7fc70d443640 1 librados: starting msgr at 2024-03-29T10:50:59.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.743+0000 7fc70d443640 1 librados: starting objecter 2024-03-29T10:50:59.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.743+0000 7fc70d443640 1 Processor -- start 2024-03-29T10:50:59.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.743+0000 7fc70d443640 1 -- start start 2024-03-29T10:50:59.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.744+0000 7fc70d443640 1 librados: setting wanted keys 2024-03-29T10:50:59.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.744+0000 7fc70d443640 1 librados: calling monclient init 2024-03-29T10:50:59.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.744+0000 7fc70d443640 10 monclient: init 2024-03-29T10:50:59.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.745+0000 7fc70d443640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:59.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.745+0000 7fc70d443640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:50:59.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.746+0000 7fc70d443640 10 monclient(hunting): picked mon.noname-c con 0x7fc7080ac5a0 addr v1:172.21.15.98:6790/0 2024-03-29T10:50:59.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.746+0000 7fc70d443640 10 monclient(hunting): picked mon.noname-a con 0x7fc7080b0770 addr v1:172.21.15.98:6789/0 2024-03-29T10:50:59.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.746+0000 7fc70d443640 10 monclient(hunting): picked mon.noname-b con 0x7fc70814a700 addr v1:172.21.15.189:6789/0 2024-03-29T10:50:59.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.746+0000 7fc70d443640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70812a530 con 0x7fc7080b0770 2024-03-29T10:50:59.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.746+0000 7fc70d443640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc708129800 con 0x7fc70814a700 2024-03-29T10:50:59.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.746+0000 7fc70d443640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70812b550 con 0x7fc7080ac5a0 2024-03-29T10:50:59.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.746+0000 7fc70d443640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:59.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.746+0000 7fc70d443640 10 monclient(hunting): authenticate will time out at 1279.943533s 2024-03-29T10:50:59.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.749+0000 7fc706ffd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fc7080ac5a0 0x7fc7081290f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:60003/0 (socket says 172.21.15.98:60003) 2024-03-29T10:50:59.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.749+0000 7fc706ffd640 1 -- 172.21.15.98:0/3436198939 learned_addr learned my addr 172.21.15.98:0/3436198939 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:50:59.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.750+0000 7fc6ef7fe640 10 client.?.objecter ms_handle_connect 0x7fc7080ac5a0 2024-03-29T10:50:59.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.750+0000 7fc6ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:59.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.750+0000 7fc6ef7fe640 10 client.?.objecter ms_handle_connect 0x7fc7080b0770 2024-03-29T10:50:59.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.750+0000 7fc6ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:59.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.752+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000490269 0 0) 0x7fc70812b550 con 0x7fc7080ac5a0 2024-03-29T10:50:59.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.752+0000 7fc6ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:59.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.752+0000 7fc6ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:59.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.752+0000 7fc6ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:59.740 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.752+0000 7fc6ef7fe640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.752+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f4004eb0 con 0x7fc7080ac5a0 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.752+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468940260 0 0) 0x7fc70812a530 con 0x7fc7080b0770 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.752+0000 7fc6ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.752+0000 7fc6ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.752+0000 7fc6ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.752+0000 7fc6ef7fe640 10 monclient(hunting): my global_id is 4446 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.752+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc70812b550 con 0x7fc7080b0770 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 10 client.?.objecter ms_handle_connect 0x7fc70814a700 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991379078 0 0) 0x7fc708129800 con 0x7fc70814a700 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:59.741 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 10 monclient(hunting): my global_id is 4453 2024-03-29T10:50:59.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc70812a530 con 0x7fc70814a700 2024-03-29T10:50:59.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 625372610 0 0) 0x7fc70812b550 con 0x7fc7080b0770 2024-03-29T10:50:59.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc708129800 con 0x7fc7080b0770 2024-03-29T10:50:59.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fc6f8001850 con 0x7fc7080b0770 2024-03-29T10:50:59.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:59.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:59.742 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 10 monclient(hunting): dump: 2024-03-29T10:50:59.742 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:59.742 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:59.742 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:59.742 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:59.742 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:59.742 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.753+0000 7fc6ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1366404103 0 0) 0x7fc6f4004eb0 con 0x7fc7080ac5a0 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc70812b550 con 0x7fc7080ac5a0 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fc700001d90 con 0x7fc7080ac5a0 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient(hunting): dump: 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:59.743 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4117413143 0 0) 0x7fc70812a530 con 0x7fc70814a700 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6f4004eb0 con 0x7fc70814a700 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fc6fc001960 con 0x7fc70814a700 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient(hunting): dump: 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:59.744 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1765249315 0 0) 0x7fc70812b550 con 0x7fc7080ac5a0 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 >> v1:172.21.15.189:6789/0 conn(0x7fc70814a700 legacy=0x7fc7081dba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 >> v1:172.21.15.98:6789/0 conn(0x7fc7080b0770 legacy=0x7fc708147f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 1 monclient: found mon.c 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:59.745 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc708129ee0 con 0x7fc7080ac5a0 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient: _finish_auth 0 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc700001ef0 con 0x7fc7080ac5a0 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fc700003de0 con 0x7fc7080ac5a0 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc6ef7fe640 10 monclient: dump: 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:59.746 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc70d443640 5 monclient: authenticate success, global_id 4481 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc70d443640 10 monclient: _renew_subs 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc70d443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc70d443640 1 -- 172.21.15.98:0/3436198939 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70812a1b0 con 0x7fc7080ac5a0 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.754+0000 7fc70d443640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.755+0000 7fc70d443640 10 monclient: _renew_subs 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.755+0000 7fc70d443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.755+0000 7fc70d443640 1 -- 172.21.15.98:0/3436198939 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc708148850 con 0x7fc7080ac5a0 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.755+0000 7fc70d443640 1 librados: init done 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.756+0000 7fc70d443640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.756+0000 7fc70d443640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:59.747 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.756+0000 7fc70d443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:50:59.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.756+0000 7fc70d443640 1 -- 172.21.15.98:0/3436198939 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7080ab560 con 0x7fc7080ac5a0 2024-03-29T10:50:59.748 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.757+0000 7fc7057fa640 4 set_mon_vals no callback set 2024-03-29T10:50:59.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.759+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fc700003f40 con 0x7fc7080ac5a0 2024-03-29T10:50:59.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.760+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fc700084e50 con 0x7fc7080ac5a0 2024-03-29T10:50:59.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.760+0000 7fc7067fc640 10 monclient: get_auth_request con 0x7fc6f406e120 auth_method 0 2024-03-29T10:50:59.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.760+0000 7fc6ef7fe640 10 client.4481.objecter ms_dispatch 0x7fc708000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:50:59.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.760+0000 7fc6ef7fe640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:50:59.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.760+0000 7fc6ef7fe640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:50:59.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.760+0000 7fc6ef7fe640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T10:50:59.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.761+0000 7fc6ef7fe640 10 client.4481.objecter ms_handle_connect 0x7fc6f406e120 2024-03-29T10:50:59.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.766+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc700050730 con 0x7fc7080ac5a0 2024-03-29T10:50:59.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.766+0000 7fc6ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:59.749 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:50:59.766+0000 7fc6ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:00.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.147+0000 7fc70d443640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:00.129 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.147+0000 7fc70d443640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:00.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.147+0000 7fc70d443640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:00.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.147+0000 7fc70d443640 1 -- 172.21.15.98:0/3436198939 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc708075f10 con 0x7fc7080ac5a0 2024-03-29T10:51:00.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.148+0000 7fc6ef7fe640 1 -- 172.21.15.98:0/3436198939 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fc700052660 con 0x7fc7080ac5a0 2024-03-29T10:51:00.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.148+0000 7fc6ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:00.130 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.148+0000 7fc6ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:00.131 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:51:00.131 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:00.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.152+0000 7fc70d443640 10 librados: watch_flush enter 2024-03-29T10:51:00.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.152+0000 7fc70d443640 10 librados: watch_flush exit 2024-03-29T10:51:00.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.152+0000 7fc70d443640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T10:51:00.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.152+0000 7fc70d443640 10 client.4481.objecter successfully canceled tick 2024-03-29T10:51:00.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.152+0000 7fc70d443640 1 -- 172.21.15.98:0/3436198939 >> v1:172.21.15.98:6816/1481799426 conn(0x7fc6f406e120 legacy=0x7fc6f40705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:00.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.152+0000 7fc70d443640 10 monclient: shutdown 2024-03-29T10:51:00.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.152+0000 7fc70d443640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:00.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.152+0000 7fc70d443640 1 -- 172.21.15.98:0/3436198939 >> v1:172.21.15.98:6790/0 conn(0x7fc7080ac5a0 legacy=0x7fc7081290f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:00.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.152+0000 7fc70d443640 1 -- 172.21.15.98:0/3436198939 shutdown_connections 2024-03-29T10:51:00.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.152+0000 7fc70d443640 1 -- 172.21.15.98:0/3436198939 >> 172.21.15.98:0/3436198939 conn(0x7fc70806b660 msgr2=0x7fc708149530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:00.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.153+0000 7fc70d443640 1 -- 172.21.15.98:0/3436198939 shutdown_connections 2024-03-29T10:51:00.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.153+0000 7fc70d443640 1 -- 172.21.15.98:0/3436198939 wait complete. 2024-03-29T10:51:00.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:00.153+0000 7fc70d443640 1 librados: shutdown 2024-03-29T10:51:00.149 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:06.956 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:07.115 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.133+0000 7f3b640fb640 10 monclient: get_monmap_and_config 2024-03-29T10:51:07.115 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.133+0000 7f3b640fb640 10 monclient: build_initial_monmap 2024-03-29T10:51:07.115 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.133+0000 7f3b640fb640 10 monclient: monmap: 2024-03-29T10:51:07.115 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:07.115 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:07.115 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:51:07.134597+0000 2024-03-29T10:51:07.115 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:51:07.134597+0000 2024-03-29T10:51:07.115 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:07.116 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:07.116 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:07.116 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:51:07.116 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:51:07.116 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:07.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.137+0000 7f3b640fb640 1 Processor -- start 2024-03-29T10:51:07.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.138+0000 7f3b640fb640 1 -- start start 2024-03-29T10:51:07.119 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.138+0000 7f3b640fb640 10 monclient: init 2024-03-29T10:51:07.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.139+0000 7f3b640fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:07.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.139+0000 7f3b640fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:07.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.139+0000 7f3b640fb640 10 monclient(hunting): picked mon.noname-c con 0x7f3b5c1457f0 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:07.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.139+0000 7f3b640fb640 10 monclient(hunting): picked mon.noname-a con 0x7f3b5c13f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:07.120 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.139+0000 7f3b640fb640 10 monclient(hunting): picked mon.noname-b con 0x7f3b5c0a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:07.121 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.139+0000 7f3b640fb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c0aaef0 con 0x7f3b5c13f650 2024-03-29T10:51:07.121 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.139+0000 7f3b640fb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c145bd0 con 0x7f3b5c1457f0 2024-03-29T10:51:07.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.142+0000 7f3b640fb640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c145eb0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.142+0000 7f3b640fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:07.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.142+0000 7f3b640fb640 10 monclient(hunting): authenticate will time out at 1287.339380s 2024-03-29T10:51:07.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.145+0000 7f3b61696640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3b5c13f650 0x7f3b5c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47295/0 (socket says 172.21.15.98:47295) 2024-03-29T10:51:07.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.145+0000 7f3b61696640 1 -- 172.21.15.98:0/506770864 learned_addr learned my addr 172.21.15.98:0/506770864 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:07.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 1 -- 172.21.15.98:0/506770864 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765865426 0 0) 0x7f3b5c0aaef0 con 0x7f3b5c13f650 2024-03-29T10:51:07.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 10 monclient(hunting): _negotiate 2024-03-29T10:51:07.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:07.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:07.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 10 monclient(hunting): my global_id is 4449 2024-03-29T10:51:07.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 1 -- 172.21.15.98:0/506770864 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b48003520 con 0x7f3b5c13f650 2024-03-29T10:51:07.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 1 -- 172.21.15.98:0/506770864 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846192666 0 0) 0x7f3b5c145eb0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 10 monclient(hunting): _negotiate 2024-03-29T10:51:07.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:07.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:07.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 10 monclient(hunting): my global_id is 4484 2024-03-29T10:51:07.131 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 1 -- 172.21.15.98:0/506770864 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b5c0aaef0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 1 -- 172.21.15.98:0/506770864 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185143101 0 0) 0x7f3b5c145bd0 con 0x7f3b5c1457f0 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 10 monclient(hunting): _negotiate 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 10 monclient(hunting): my global_id is 4456 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.147+0000 7f3b60e95640 1 -- 172.21.15.98:0/506770864 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b5c145eb0 con 0x7f3b5c1457f0 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 1 -- 172.21.15.98:0/506770864 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3643372682 0 0) 0x7f3b5c145eb0 con 0x7f3b5c1457f0 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 1 -- 172.21.15.98:0/506770864 >> v1:172.21.15.98:6790/0 conn(0x7f3b5c0a9d70 legacy=0x7f3b5c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 1 -- 172.21.15.98:0/506770864 >> v1:172.21.15.98:6789/0 conn(0x7f3b5c13f650 legacy=0x7f3b5c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 10 monclient: _finish_hunting 0 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 1 monclient: found mon.noname-c 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 1 -- 172.21.15.98:0/506770864 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c146340 con 0x7f3b5c1457f0 2024-03-29T10:51:07.132 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 10 monclient: _finish_auth 0 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 10 monclient: _check_auth_tickets 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b640fb640 5 monclient: authenticate success, global_id 4456 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b640fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 1 -- 172.21.15.98:0/506770864 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3b540017e0 con 0x7f3b5c1457f0 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 10 monclient: dump: 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:07.133 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b640fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 1 -- 172.21.15.98:0/506770864 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b54001a00 con 0x7f3b5c1457f0 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b640fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b640fb640 4 set_mon_vals no callback set 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 1 -- 172.21.15.98:0/506770864 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3b54001d90 con 0x7f3b5c1457f0 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.148+0000 7f3b60e95640 10 monclient: dump: 2024-03-29T10:51:07.134 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:07.135 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:07.135 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:07.135 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:07.135 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:07.135 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:07.135 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:07.135 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:07.135 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:07.135 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:07.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.149+0000 7f3b640fb640 10 monclient: shutdown 2024-03-29T10:51:07.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.149+0000 7f3b640fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:07.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.149+0000 7f3b640fb640 1 -- 172.21.15.98:0/506770864 >> v1:172.21.15.189:6789/0 conn(0x7f3b5c1457f0 legacy=0x7f3b5c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:07.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.152+0000 7f3b640fb640 1 -- 172.21.15.98:0/506770864 shutdown_connections 2024-03-29T10:51:07.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.152+0000 7f3b640fb640 1 -- 172.21.15.98:0/506770864 >> 172.21.15.98:0/506770864 conn(0x7f3b5c066a00 msgr2=0x7f3b5c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:07.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.154+0000 7f3b640fb640 1 -- 172.21.15.98:0/506770864 shutdown_connections 2024-03-29T10:51:07.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.155+0000 7f3b640fb640 1 -- 172.21.15.98:0/506770864 wait complete. 2024-03-29T10:51:07.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.156+0000 7f3b640fb640 10 monclient: build_initial_monmap 2024-03-29T10:51:07.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.156+0000 7f3b640fb640 10 monclient: monmap: 2024-03-29T10:51:07.138 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:07.138 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:07.138 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:51:07.138 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:51:07.138 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:07.138 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:07.138 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:07.139 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:51:07.139 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:51:07.139 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:07.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.156+0000 7f3b640fb640 1 librados: starting msgr at 2024-03-29T10:51:07.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.156+0000 7f3b640fb640 1 librados: starting objecter 2024-03-29T10:51:07.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.156+0000 7f3b640fb640 1 Processor -- start 2024-03-29T10:51:07.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.156+0000 7f3b640fb640 1 -- start start 2024-03-29T10:51:07.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.156+0000 7f3b640fb640 1 librados: setting wanted keys 2024-03-29T10:51:07.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.156+0000 7f3b640fb640 1 librados: calling monclient init 2024-03-29T10:51:07.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.156+0000 7f3b640fb640 10 monclient: init 2024-03-29T10:51:07.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.159+0000 7f3b640fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:07.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.159+0000 7f3b640fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:07.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.159+0000 7f3b640fb640 10 monclient(hunting): picked mon.noname-b con 0x7f3b5c0a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:07.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.159+0000 7f3b640fb640 10 monclient(hunting): picked mon.noname-a con 0x7f3b5c13f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:07.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.159+0000 7f3b640fb640 10 monclient(hunting): picked mon.noname-c con 0x7f3b5c1457f0 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:07.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.159+0000 7f3b640fb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c140cd0 con 0x7f3b5c13f650 2024-03-29T10:51:07.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.159+0000 7f3b640fb640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c140fb0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.159+0000 7f3b640fb640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c141290 con 0x7f3b5c1457f0 2024-03-29T10:51:07.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.159+0000 7f3b640fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:07.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.159+0000 7f3b640fb640 10 monclient(hunting): authenticate will time out at 1287.356199s 2024-03-29T10:51:07.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.163+0000 7f3b61e97640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3b5c0a9d70 0x7f3b5c1416b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:42747/0 (socket says 172.21.15.98:42747) 2024-03-29T10:51:07.144 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.163+0000 7f3b61e97640 1 -- 172.21.15.98:0/2766568901 learned_addr learned my addr 172.21.15.98:0/2766568901 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:07.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.163+0000 7f3b527fc640 10 client.?.objecter ms_handle_connect 0x7f3b5c0a9d70 2024-03-29T10:51:07.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.163+0000 7f3b527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:07.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.164+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192961831 0 0) 0x7f3b5c140fb0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.164+0000 7f3b527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:07.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.164+0000 7f3b527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:07.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.164+0000 7f3b527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:07.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.164+0000 7f3b527fc640 10 monclient(hunting): my global_id is 4459 2024-03-29T10:51:07.145 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.164+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b40004eb0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 10 client.?.objecter ms_handle_connect 0x7f3b5c1457f0 2024-03-29T10:51:07.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:07.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 154792954 0 0) 0x7f3b40004eb0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b5c140fb0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106046527 0 0) 0x7f3b5c141290 con 0x7f3b5c1457f0 2024-03-29T10:51:07.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:07.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:07.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:07.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:51:07.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b40004eb0 con 0x7f3b5c1457f0 2024-03-29T10:51:07.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3b54001750 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.149 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:07.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:07.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 10 monclient(hunting): dump: 2024-03-29T10:51:07.150 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:07.150 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:07.150 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:07.150 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:07.150 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:07.150 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:07.150 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:07.150 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:07.150 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:07.150 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:07.150 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.166+0000 7f3b527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b527fc640 10 client.?.objecter ms_handle_connect 0x7f3b5c13f650 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2130116844 0 0) 0x7f3b5c140fb0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 >> v1:172.21.15.98:6790/0 conn(0x7f3b5c1457f0 legacy=0x7f3b5c1ef240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 >> v1:172.21.15.98:6789/0 conn(0x7f3b5c13f650 legacy=0x7f3b5c141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b527fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b527fc640 1 monclient: found mon.b 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b527fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c144ee0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b527fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b527fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:07.151 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b640fb640 5 monclient: authenticate success, global_id 4459 2024-03-29T10:51:07.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b640fb640 10 monclient: _renew_subs 2024-03-29T10:51:07.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b640fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:07.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b640fb640 1 -- 172.21.15.98:0/2766568901 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5c13fda0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b640fb640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:07.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b640fb640 10 monclient: _renew_subs 2024-03-29T10:51:07.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b640fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:07.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b640fb640 1 -- 172.21.15.98:0/2766568901 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5c13df70 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.167+0000 7f3b640fb640 1 librados: init done 2024-03-29T10:51:07.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.168+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b54003c20 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.168+0000 7f3b527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:07.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.168+0000 7f3b53fff640 4 set_mon_vals no callback set 2024-03-29T10:51:07.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.169+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3b54003ff0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.169+0000 7f3b527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:07.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.169+0000 7f3b527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.169+0000 7f3b527fc640 10 monclient: dump: 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.171+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f3b54004150 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.171+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f3b54084eb0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.154 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.171+0000 7f3b527fc640 10 client.4459.objecter ms_dispatch 0x7f3b5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:07.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.171+0000 7f3b527fc640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:07.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.171+0000 7f3b527fc640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:07.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.171+0000 7f3b527fc640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-29T10:51:07.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.172+0000 7f3b61696640 10 monclient: get_auth_request con 0x7f3b4006de50 auth_method 0 2024-03-29T10:51:07.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.172+0000 7f3b527fc640 10 client.4459.objecter ms_handle_connect 0x7f3b4006de50 2024-03-29T10:51:07.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.173+0000 7f3b640fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:07.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.173+0000 7f3b640fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:07.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.173+0000 7f3b640fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:07.155 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.173+0000 7f3b640fb640 1 -- 172.21.15.98:0/2766568901 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b44005080 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.181+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b54050790 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.181+0000 7f3b527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:07.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.181+0000 7f3b527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:07.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.539+0000 7f3b640fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:07.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.539+0000 7f3b640fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:07.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.539+0000 7f3b640fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:07.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.539+0000 7f3b640fb640 1 -- 172.21.15.98:0/2766568901 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b44005330 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.544+0000 7f3b527fc640 1 -- 172.21.15.98:0/2766568901 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f3b540526c0 con 0x7f3b5c0a9d70 2024-03-29T10:51:07.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.544+0000 7f3b527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:07.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.544+0000 7f3b527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:07.526 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:51:07.526 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:07.530 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.547+0000 7f3b640fb640 10 librados: watch_flush enter 2024-03-29T10:51:07.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.549+0000 7f3b640fb640 10 librados: watch_flush exit 2024-03-29T10:51:07.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.549+0000 7f3b640fb640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-29T10:51:07.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.549+0000 7f3b640fb640 10 client.4459.objecter successfully canceled tick 2024-03-29T10:51:07.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.550+0000 7f3b640fb640 1 -- 172.21.15.98:0/2766568901 >> v1:172.21.15.98:6816/1481799426 conn(0x7f3b4006de50 legacy=0x7f3b40070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:07.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.550+0000 7f3b640fb640 10 monclient: shutdown 2024-03-29T10:51:07.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.550+0000 7f3b640fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:07.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.550+0000 7f3b640fb640 1 -- 172.21.15.98:0/2766568901 >> v1:172.21.15.189:6789/0 conn(0x7f3b5c0a9d70 legacy=0x7f3b5c1416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:07.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.550+0000 7f3b640fb640 1 -- 172.21.15.98:0/2766568901 shutdown_connections 2024-03-29T10:51:07.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.550+0000 7f3b640fb640 1 -- 172.21.15.98:0/2766568901 >> 172.21.15.98:0/2766568901 conn(0x7f3b5c066a00 msgr2=0x7f3b5c1daae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:07.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.552+0000 7f3b640fb640 1 -- 172.21.15.98:0/2766568901 shutdown_connections 2024-03-29T10:51:07.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.553+0000 7f3b640fb640 1 -- 172.21.15.98:0/2766568901 wait complete. 2024-03-29T10:51:07.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:07.553+0000 7f3b640fb640 1 librados: shutdown 2024-03-29T10:51:07.557 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:14.364 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:14.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.533+0000 7f5c9b863640 10 monclient: get_monmap_and_config 2024-03-29T10:51:14.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.533+0000 7f5c9b863640 10 monclient: build_initial_monmap 2024-03-29T10:51:14.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.533+0000 7f5c9b863640 10 monclient: monmap: 2024-03-29T10:51:14.515 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:14.515 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:14.515 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:51:14.533988+0000 2024-03-29T10:51:14.515 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:51:14.533988+0000 2024-03-29T10:51:14.515 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:14.515 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:14.515 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:14.515 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:51:14.515 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:51:14.516 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:14.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.534+0000 7f5c9b863640 1 Processor -- start 2024-03-29T10:51:14.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.535+0000 7f5c9b863640 1 -- start start 2024-03-29T10:51:14.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.535+0000 7f5c9b863640 10 monclient: init 2024-03-29T10:51:14.519 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.535+0000 7f5c9b863640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:14.519 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.536+0000 7f5c9b863640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:14.519 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.536+0000 7f5c9b863640 10 monclient(hunting): picked mon.noname-b con 0x7f5c9406cac0 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:14.519 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.536+0000 7f5c9b863640 10 monclient(hunting): picked mon.noname-c con 0x7f5c9406cea0 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:14.519 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.536+0000 7f5c9b863640 10 monclient(hunting): picked mon.noname-a con 0x7f5c94145e80 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:14.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.536+0000 7f5c995ff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f5c9406cac0 0x7f5c94071ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:58954/0 (socket says 172.21.15.98:58954) 2024-03-29T10:51:14.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.536+0000 7f5c995ff640 1 -- 172.21.15.98:0/2685904427 learned_addr learned my addr 172.21.15.98:0/2685904427 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:14.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.536+0000 7f5c9b863640 1 -- 172.21.15.98:0/2685904427 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c940ac850 con 0x7f5c94145e80 2024-03-29T10:51:14.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.536+0000 7f5c9b863640 1 -- 172.21.15.98:0/2685904427 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c94070470 con 0x7f5c9406cea0 2024-03-29T10:51:14.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.536+0000 7f5c9b863640 1 -- 172.21.15.98:0/2685904427 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c94072600 con 0x7f5c9406cac0 2024-03-29T10:51:14.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.536+0000 7f5c9b863640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:14.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.536+0000 7f5c9b863640 10 monclient(hunting): authenticate will time out at 1294.733425s 2024-03-29T10:51:14.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.536+0000 7f5c8bfff640 1 -- 172.21.15.98:0/2685904427 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529006822 0 0) 0x7f5c94072600 con 0x7f5c9406cac0 2024-03-29T10:51:14.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:14.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:14.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:14.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:51:14.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 1 -- 172.21.15.98:0/2685904427 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c80003520 con 0x7f5c9406cac0 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 1 -- 172.21.15.98:0/2685904427 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387854369 0 0) 0x7f5c94070470 con 0x7f5c9406cea0 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient(hunting): my global_id is 4462 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 1 -- 172.21.15.98:0/2685904427 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c94072600 con 0x7f5c9406cea0 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 1 -- 172.21.15.98:0/2685904427 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393911203 0 0) 0x7f5c940ac850 con 0x7f5c94145e80 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient(hunting): my global_id is 4455 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 1 -- 172.21.15.98:0/2685904427 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c94070470 con 0x7f5c94145e80 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 1 -- 172.21.15.98:0/2685904427 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4157850079 0 0) 0x7f5c94070470 con 0x7f5c94145e80 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 1 -- 172.21.15.98:0/2685904427 >> v1:172.21.15.98:6790/0 conn(0x7f5c9406cac0 legacy=0x7f5c94071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:14.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 1 -- 172.21.15.98:0/2685904427 >> v1:172.21.15.189:6789/0 conn(0x7f5c9406cea0 legacy=0x7f5c94145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 1 monclient: found mon.noname-a 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 1 -- 172.21.15.98:0/2685904427 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c941462d0 con 0x7f5c94145e80 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient: _finish_auth 0 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 1 -- 172.21.15.98:0/2685904427 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5c8c001930 con 0x7f5c94145e80 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c8bfff640 10 monclient: dump: 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:14.522 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c9b863640 5 monclient: authenticate success, global_id 4455 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.537+0000 7f5c9b863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.538+0000 7f5c8bfff640 1 -- 172.21.15.98:0/2685904427 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c8c001a90 con 0x7f5c94145e80 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.538+0000 7f5c8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.538+0000 7f5c8bfff640 1 -- 172.21.15.98:0/2685904427 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5c8c001ea0 con 0x7f5c94145e80 2024-03-29T10:51:14.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.538+0000 7f5c8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.538+0000 7f5c8bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.538+0000 7f5c8bfff640 10 monclient: dump: 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.538+0000 7f5c9b863640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:14.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.538+0000 7f5c9b863640 4 set_mon_vals no callback set 2024-03-29T10:51:14.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.539+0000 7f5c9b863640 10 monclient: shutdown 2024-03-29T10:51:14.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.539+0000 7f5c9b863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:14.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.539+0000 7f5c9b863640 1 -- 172.21.15.98:0/2685904427 >> v1:172.21.15.98:6789/0 conn(0x7f5c94145e80 legacy=0x7f5c9414db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:14.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.539+0000 7f5c9b863640 1 -- 172.21.15.98:0/2685904427 shutdown_connections 2024-03-29T10:51:14.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.539+0000 7f5c9b863640 1 -- 172.21.15.98:0/2685904427 >> 172.21.15.98:0/2685904427 conn(0x7f5c940675b0 msgr2=0x7f5c940b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:14.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.539+0000 7f5c9b863640 1 -- 172.21.15.98:0/2685904427 shutdown_connections 2024-03-29T10:51:14.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.539+0000 7f5c9b863640 1 -- 172.21.15.98:0/2685904427 wait complete. 2024-03-29T10:51:14.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.540+0000 7f5c9b863640 10 monclient: build_initial_monmap 2024-03-29T10:51:14.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.540+0000 7f5c9b863640 10 monclient: monmap: 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.541+0000 7f5c9b863640 1 librados: starting msgr at 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.541+0000 7f5c9b863640 1 librados: starting objecter 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.541+0000 7f5c9b863640 1 Processor -- start 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.541+0000 7f5c9b863640 1 -- start start 2024-03-29T10:51:14.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.541+0000 7f5c9b863640 1 librados: setting wanted keys 2024-03-29T10:51:14.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.541+0000 7f5c9b863640 1 librados: calling monclient init 2024-03-29T10:51:14.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.541+0000 7f5c9b863640 10 monclient: init 2024-03-29T10:51:14.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.542+0000 7f5c9b863640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:14.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.542+0000 7f5c9b863640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:14.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.542+0000 7f5c9b863640 10 monclient(hunting): picked mon.noname-b con 0x7f5c9406cac0 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:14.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c9b863640 10 monclient(hunting): picked mon.noname-c con 0x7f5c9406cea0 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:14.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c98dfe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f5c9406cea0 0x7f5c941373f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:59386/0 (socket says 172.21.15.98:59386) 2024-03-29T10:51:14.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c995ff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5c9406cac0 0x7f5c940ab950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:51532/0 (socket says 172.21.15.98:51532) 2024-03-29T10:51:14.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c98dfe640 1 -- 172.21.15.98:0/3858028832 learned_addr learned my addr 172.21.15.98:0/3858028832 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:14.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c99e00640 1 --1- 172.21.15.98:0/3858028832 >> v1:172.21.15.98:6789/0 conn(0x7f5c94145e80 0x7f5c94137b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:34238/0 (socket says 172.21.15.98:34238) 2024-03-29T10:51:14.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c9b863640 10 monclient(hunting): picked mon.noname-a con 0x7f5c94145e80 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:14.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c9b863640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c94138210 con 0x7f5c94145e80 2024-03-29T10:51:14.534 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c89ffb640 10 client.?.objecter ms_handle_connect 0x7f5c9406cea0 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c89ffb640 10 client.?.objecter ms_handle_connect 0x7f5c94145e80 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c89ffb640 10 client.?.objecter ms_handle_connect 0x7f5c9406cac0 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c9b863640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c94139fe0 con 0x7f5c9406cac0 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c9b863640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c9413a170 con 0x7f5c9406cea0 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c9b863640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.543+0000 7f5c9b863640 10 monclient(hunting): authenticate will time out at 1294.740816s 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623779484 0 0) 0x7f5c9413a170 con 0x7f5c9406cea0 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:51:14.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c78004eb0 con 0x7f5c9406cea0 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059879340 0 0) 0x7f5c94138210 con 0x7f5c94145e80 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c9413a170 con 0x7f5c94145e80 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505178869 0 0) 0x7f5c94139fe0 con 0x7f5c9406cac0 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 10 monclient(hunting): my global_id is 4465 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.544+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c94138210 con 0x7f5c9406cac0 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 609434987 0 0) 0x7f5c9413a170 con 0x7f5c94145e80 2024-03-29T10:51:14.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c94139fe0 con 0x7f5c94145e80 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5c8c001da0 con 0x7f5c94145e80 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 10 monclient(hunting): dump: 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:14.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2868554515 0 0) 0x7f5c94138210 con 0x7f5c9406cac0 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c9413a170 con 0x7f5c9406cac0 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5c84001890 con 0x7f5c9406cac0 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 10 monclient(hunting): dump: 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:14.538 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 918864581 0 0) 0x7f5c78004eb0 con 0x7f5c9406cea0 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c94138210 con 0x7f5c9406cea0 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1981903493 0 0) 0x7f5c9413a170 con 0x7f5c9406cac0 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 >> v1:172.21.15.98:6790/0 conn(0x7f5c9406cea0 legacy=0x7f5c941373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 >> v1:172.21.15.98:6789/0 conn(0x7f5c94145e80 legacy=0x7f5c94137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 1 monclient: found mon.b 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9413a300 con 0x7f5c9406cac0 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:14.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:14.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c9b863640 5 monclient: authenticate success, global_id 4465 2024-03-29T10:51:14.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c9b863640 10 monclient: _renew_subs 2024-03-29T10:51:14.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c9b863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:14.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.545+0000 7f5c9b863640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c940aa140 con 0x7f5c9406cac0 2024-03-29T10:51:14.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.546+0000 7f5c9b863640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:14.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.546+0000 7f5c9b863640 10 monclient: _renew_subs 2024-03-29T10:51:14.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.546+0000 7f5c9b863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:14.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.546+0000 7f5c9b863640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c940a94e0 con 0x7f5c9406cac0 2024-03-29T10:51:14.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.546+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c84001ca0 con 0x7f5c9406cac0 2024-03-29T10:51:14.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.546+0000 7f5c89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:14.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.546+0000 7f5c8b7fe640 4 set_mon_vals no callback set 2024-03-29T10:51:14.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.546+0000 7f5c9b863640 1 librados: init done 2024-03-29T10:51:14.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.546+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5c84004980 con 0x7f5c9406cac0 2024-03-29T10:51:14.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.546+0000 7f5c89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:14.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.546+0000 7f5c89ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:14.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.546+0000 7f5c89ffb640 10 monclient: dump: 2024-03-29T10:51:14.542 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:14.542 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:14.542 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:14.542 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:14.542 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:14.542 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.547+0000 7f5c7f7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.547+0000 7f5c7f7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.547+0000 7f5c7f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.547+0000 7f5c7f7fe640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c6c005080 con 0x7f5c9406cac0 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.548+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f5c8401a950 con 0x7f5c9406cac0 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.549+0000 7f5c98dfe640 10 monclient: get_auth_request con 0x7f5c7806e200 auth_method 0 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.549+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f5c84084780 con 0x7f5c9406cac0 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.549+0000 7f5c89ffb640 10 client.4465.objecter ms_dispatch 0x7f5c94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.549+0000 7f5c89ffb640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:14.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.549+0000 7f5c89ffb640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:14.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.550+0000 7f5c89ffb640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-29T10:51:14.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.550+0000 7f5c89ffb640 10 client.4465.objecter ms_handle_connect 0x7f5c7806e200 2024-03-29T10:51:14.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.557+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c8404fe90 con 0x7f5c9406cac0 2024-03-29T10:51:14.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.557+0000 7f5c89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:14.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.557+0000 7f5c89ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:14.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.912+0000 7f5c7f7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:14.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.912+0000 7f5c7f7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:14.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.912+0000 7f5c7f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:14.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.912+0000 7f5c7f7fe640 1 -- 172.21.15.98:0/3858028832 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c6c005330 con 0x7f5c9406cac0 2024-03-29T10:51:14.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.915+0000 7f5c89ffb640 1 -- 172.21.15.98:0/3858028832 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f5c84051f90 con 0x7f5c9406cac0 2024-03-29T10:51:14.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.915+0000 7f5c89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:14.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.915+0000 7f5c89ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:14.897 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:51:14.897 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:14.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.919+0000 7f5c7f7fe640 10 librados: watch_flush enter 2024-03-29T10:51:14.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.919+0000 7f5c7f7fe640 10 librados: watch_flush exit 2024-03-29T10:51:14.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.919+0000 7f5c7f7fe640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-29T10:51:14.900 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.919+0000 7f5c7f7fe640 10 client.4465.objecter successfully canceled tick 2024-03-29T10:51:14.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.919+0000 7f5c7f7fe640 1 -- 172.21.15.98:0/3858028832 >> v1:172.21.15.98:6816/1481799426 conn(0x7f5c7806e200 legacy=0x7f5c780706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:14.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.919+0000 7f5c7f7fe640 10 monclient: shutdown 2024-03-29T10:51:14.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.919+0000 7f5c7f7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:14.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.919+0000 7f5c7f7fe640 1 -- 172.21.15.98:0/3858028832 >> v1:172.21.15.189:6789/0 conn(0x7f5c9406cac0 legacy=0x7f5c940ab950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:14.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.919+0000 7f5c7f7fe640 1 -- 172.21.15.98:0/3858028832 shutdown_connections 2024-03-29T10:51:14.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.919+0000 7f5c7f7fe640 1 -- 172.21.15.98:0/3858028832 >> 172.21.15.98:0/3858028832 conn(0x7f5c940675b0 msgr2=0x7f5c9413f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:14.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.919+0000 7f5c7f7fe640 1 -- 172.21.15.98:0/3858028832 shutdown_connections 2024-03-29T10:51:14.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.919+0000 7f5c7f7fe640 1 -- 172.21.15.98:0/3858028832 wait complete. 2024-03-29T10:51:14.901 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:14.920+0000 7f5c7f7fe640 1 librados: shutdown 2024-03-29T10:51:14.917 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:21.724 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:21.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.871+0000 7f702fadd640 10 monclient: get_monmap_and_config 2024-03-29T10:51:21.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.871+0000 7f702fadd640 10 monclient: build_initial_monmap 2024-03-29T10:51:21.854 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.871+0000 7f702fadd640 10 monclient: monmap: 2024-03-29T10:51:21.854 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:21.854 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:21.854 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:51:21.873480+0000 2024-03-29T10:51:21.854 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:51:21.873480+0000 2024-03-29T10:51:21.854 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:21.854 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:21.854 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:21.855 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:51:21.855 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:51:21.855 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:21.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.872+0000 7f702fadd640 1 Processor -- start 2024-03-29T10:51:21.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.872+0000 7f702fadd640 1 -- start start 2024-03-29T10:51:21.855 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.872+0000 7f702fadd640 10 monclient: init 2024-03-29T10:51:21.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.873+0000 7f702fadd640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:21.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.873+0000 7f702fadd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:21.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.873+0000 7f702fadd640 10 monclient(hunting): picked mon.noname-a con 0x7f70281457f0 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:21.856 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.873+0000 7f702fadd640 10 monclient(hunting): picked mon.noname-c con 0x7f702813f650 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.873+0000 7f702fadd640 10 monclient(hunting): picked mon.noname-b con 0x7f70280a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.873+0000 7f702fadd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70280aaef0 con 0x7f70281457f0 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.873+0000 7f702fadd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7028145bd0 con 0x7f702813f650 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.873+0000 7f702fadd640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7028145eb0 con 0x7f70280a9d70 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.873+0000 7f702fadd640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.873+0000 7f702fadd640 10 monclient(hunting): authenticate will time out at 1302.071845s 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.874+0000 7f702e07a640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f70280a9d70 0x7f702814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:38784/0 (socket says 172.21.15.98:38784) 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.874+0000 7f702e07a640 1 -- 172.21.15.98:0/2331043583 learned_addr learned my addr 172.21.15.98:0/2331043583 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.874+0000 7f702c877640 1 -- 172.21.15.98:0/2331043583 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254277437 0 0) 0x7f7028145eb0 con 0x7f70280a9d70 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.874+0000 7f702c877640 10 monclient(hunting): _negotiate 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.874+0000 7f702c877640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.874+0000 7f702c877640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.874+0000 7f702c877640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:51:21.857 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.874+0000 7f702c877640 1 -- 172.21.15.98:0/2331043583 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7014003520 con 0x7f70280a9d70 2024-03-29T10:51:21.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.874+0000 7f702c877640 1 -- 172.21.15.98:0/2331043583 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1163722216 0 0) 0x7f7014003520 con 0x7f70280a9d70 2024-03-29T10:51:21.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.874+0000 7f702c877640 1 -- 172.21.15.98:0/2331043583 >> v1:172.21.15.189:6789/0 conn(0x7f702813f650 legacy=0x7f70280b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:51:21.858 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.874+0000 7f702c877640 1 -- 172.21.15.98:0/2331043583 >> v1:172.21.15.98:6789/0 conn(0x7f70281457f0 legacy=0x7f702813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:21.860 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 10 monclient: _finish_hunting 0 2024-03-29T10:51:21.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 1 monclient: found mon.noname-b 2024-03-29T10:51:21.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:21.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:51:21.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 1 -- 172.21.15.98:0/2331043583 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7028146340 con 0x7f70280a9d70 2024-03-29T10:51:21.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 10 monclient: _finish_auth 0 2024-03-29T10:51:21.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 10 monclient: _check_auth_tickets 2024-03-29T10:51:21.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:21.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 1 -- 172.21.15.98:0/2331043583 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f7020001a10 con 0x7f70280a9d70 2024-03-29T10:51:21.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702fadd640 5 monclient: authenticate success, global_id 4496 2024-03-29T10:51:21.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702fadd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:21.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:21.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:21.861 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 10 monclient: dump: 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702fadd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 1 -- 172.21.15.98:0/2331043583 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7020001ca0 con 0x7f70280a9d70 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702c877640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:21.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702fadd640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:21.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.876+0000 7f702fadd640 4 set_mon_vals no callback set 2024-03-29T10:51:21.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.877+0000 7f702fadd640 10 monclient: shutdown 2024-03-29T10:51:21.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.877+0000 7f702fadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:21.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.877+0000 7f702fadd640 1 -- 172.21.15.98:0/2331043583 >> v1:172.21.15.98:6790/0 conn(0x7f70280a9d70 legacy=0x7f702814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:21.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.878+0000 7f702fadd640 1 -- 172.21.15.98:0/2331043583 shutdown_connections 2024-03-29T10:51:21.863 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.878+0000 7f702fadd640 1 -- 172.21.15.98:0/2331043583 >> 172.21.15.98:0/2331043583 conn(0x7f7028066a00 msgr2=0x7f7028066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:21.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.879+0000 7f702fadd640 1 -- 172.21.15.98:0/2331043583 shutdown_connections 2024-03-29T10:51:21.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.879+0000 7f702fadd640 1 -- 172.21.15.98:0/2331043583 wait complete. 2024-03-29T10:51:21.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.879+0000 7f702fadd640 10 monclient: build_initial_monmap 2024-03-29T10:51:21.866 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.879+0000 7f702fadd640 10 monclient: monmap: 2024-03-29T10:51:21.866 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:21.866 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:21.866 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:51:21.866 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:51:21.866 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:21.866 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.880+0000 7f702fadd640 1 librados: starting msgr at 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.880+0000 7f702fadd640 1 librados: starting objecter 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.880+0000 7f702fadd640 1 Processor -- start 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.880+0000 7f702fadd640 1 -- start start 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.880+0000 7f702fadd640 1 librados: setting wanted keys 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.880+0000 7f702fadd640 1 librados: calling monclient init 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.880+0000 7f702fadd640 10 monclient: init 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f702fadd640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f702fadd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:21.867 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f702fadd640 10 monclient(hunting): picked mon.noname-c con 0x7f70280a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f702fadd640 10 monclient(hunting): picked mon.noname-b con 0x7f702813f650 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f702fadd640 10 monclient(hunting): picked mon.noname-a con 0x7f70281457f0 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f702fadd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70281427e0 con 0x7f70281457f0 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f702fadd640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7028142e50 con 0x7f702813f650 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f702fadd640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7028141680 con 0x7f70280a9d70 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f702fadd640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f702fadd640 10 monclient(hunting): authenticate will time out at 1302.079107s 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f702d879640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f70280a9d70 0x7f702813b270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:39918/0 (socket says 172.21.15.98:39918) 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f702d879640 1 -- 172.21.15.98:0/58683124 learned_addr learned my addr 172.21.15.98:0/58683124 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f701e7fc640 10 client.?.objecter ms_handle_connect 0x7f70280a9d70 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f701e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751930467 0 0) 0x7f7028141680 con 0x7f70280a9d70 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f701e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f701e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:21.868 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f701e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f701e7fc640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.881+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f700c004eb0 con 0x7f70280a9d70 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.882+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1363114611 0 0) 0x7f700c004eb0 con 0x7f70280a9d70 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.882+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7028141680 con 0x7f70280a9d70 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.882+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f7024003fe0 con 0x7f70280a9d70 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.882+0000 7f701e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.882+0000 7f701e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.882+0000 7f701e7fc640 10 monclient(hunting): dump: 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:21.869 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:21.870 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:21.870 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:21.870 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:21.870 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:21.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.882+0000 7f701e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:21.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.882+0000 7f701e7fc640 10 client.?.objecter ms_handle_connect 0x7f702813f650 2024-03-29T10:51:21.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.882+0000 7f701e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:21.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.882+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1737159434 0 0) 0x7f7028141680 con 0x7f70280a9d70 2024-03-29T10:51:21.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.882+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 >> v1:172.21.15.189:6789/0 conn(0x7f702813f650 legacy=0x7f70281420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:21.870 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.883+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 >> v1:172.21.15.98:6789/0 conn(0x7f70281457f0 legacy=0x7f70281d7230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:21.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f701e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:21.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f701e7fc640 1 monclient: found mon.c 2024-03-29T10:51:21.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f701e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:21.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f701e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:21.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7028141960 con 0x7f70280a9d70 2024-03-29T10:51:21.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f701e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:21.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f701e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:21.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f701e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:21.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f702fadd640 5 monclient: authenticate success, global_id 4499 2024-03-29T10:51:21.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f702fadd640 10 monclient: _renew_subs 2024-03-29T10:51:21.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f702fadd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:21.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f702fadd640 1 -- 172.21.15.98:0/58683124 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7028140a60 con 0x7f70280a9d70 2024-03-29T10:51:21.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f702fadd640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:21.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f702fadd640 10 monclient: _renew_subs 2024-03-29T10:51:21.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f702fadd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:21.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f702fadd640 1 -- 172.21.15.98:0/58683124 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f702813ffc0 con 0x7f70280a9d70 2024-03-29T10:51:21.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f702fadd640 1 librados: init done 2024-03-29T10:51:21.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7024004140 con 0x7f70280a9d70 2024-03-29T10:51:21.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.884+0000 7f701e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:21.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.885+0000 7f701ffff640 4 set_mon_vals no callback set 2024-03-29T10:51:21.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.885+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f7024001130 con 0x7f70280a9d70 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.885+0000 7f701e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.885+0000 7f701e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.885+0000 7f701e7fc640 10 monclient: dump: 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:21.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.886+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f7024001390 con 0x7f70280a9d70 2024-03-29T10:51:21.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.886+0000 7f702fadd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:21.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.886+0000 7f702fadd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:21.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.887+0000 7f702fadd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:21.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.887+0000 7f702fadd640 1 -- 172.21.15.98:0/58683124 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7010005080 con 0x7f70280a9d70 2024-03-29T10:51:21.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.887+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f7024084a90 con 0x7f70280a9d70 2024-03-29T10:51:21.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.887+0000 7f702d078640 10 monclient: get_auth_request con 0x7f700c06dc50 auth_method 0 2024-03-29T10:51:21.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.887+0000 7f701e7fc640 10 client.4499.objecter ms_dispatch 0x7f7028000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:21.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.887+0000 7f701e7fc640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:21.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.887+0000 7f701e7fc640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:21.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.887+0000 7f701e7fc640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T10:51:21.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.887+0000 7f701e7fc640 10 client.4499.objecter ms_handle_connect 0x7f700c06dc50 2024-03-29T10:51:21.877 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.895+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7024050370 con 0x7f70280a9d70 2024-03-29T10:51:21.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.895+0000 7f701e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:21.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:21.895+0000 7f701e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:22.190 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.207+0000 7f702fadd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:22.190 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.207+0000 7f702fadd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:22.190 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.207+0000 7f702fadd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:22.190 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.207+0000 7f702fadd640 1 -- 172.21.15.98:0/58683124 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7010005330 con 0x7f70280a9d70 2024-03-29T10:51:22.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.208+0000 7f701e7fc640 1 -- 172.21.15.98:0/58683124 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f7024033db0 con 0x7f70280a9d70 2024-03-29T10:51:22.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.208+0000 7f701e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:22.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.208+0000 7f701e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:22.191 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:51:22.191 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:22.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.211+0000 7f702fadd640 10 librados: watch_flush enter 2024-03-29T10:51:22.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.212+0000 7f702fadd640 10 librados: watch_flush exit 2024-03-29T10:51:22.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.212+0000 7f702fadd640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T10:51:22.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.212+0000 7f702fadd640 10 client.4499.objecter successfully canceled tick 2024-03-29T10:51:22.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.212+0000 7f702fadd640 1 -- 172.21.15.98:0/58683124 >> v1:172.21.15.98:6816/1481799426 conn(0x7f700c06dc50 legacy=0x7f700c070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:22.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.212+0000 7f702fadd640 10 monclient: shutdown 2024-03-29T10:51:22.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.212+0000 7f702fadd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:22.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.212+0000 7f702fadd640 1 -- 172.21.15.98:0/58683124 >> v1:172.21.15.98:6790/0 conn(0x7f70280a9d70 legacy=0x7f702813b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:22.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.212+0000 7f702fadd640 1 -- 172.21.15.98:0/58683124 shutdown_connections 2024-03-29T10:51:22.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.212+0000 7f702fadd640 1 -- 172.21.15.98:0/58683124 >> 172.21.15.98:0/58683124 conn(0x7f7028066a00 msgr2=0x7f70280b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:22.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.212+0000 7f702fadd640 1 -- 172.21.15.98:0/58683124 shutdown_connections 2024-03-29T10:51:22.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.212+0000 7f702fadd640 1 -- 172.21.15.98:0/58683124 wait complete. 2024-03-29T10:51:22.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:22.213+0000 7f702fadd640 1 librados: shutdown 2024-03-29T10:51:22.210 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:29.016 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:29.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.182+0000 7fb3ae4b3640 10 monclient: get_monmap_and_config 2024-03-29T10:51:29.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.182+0000 7fb3ae4b3640 10 monclient: build_initial_monmap 2024-03-29T10:51:29.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.182+0000 7fb3ae4b3640 10 monclient: monmap: 2024-03-29T10:51:29.165 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:29.165 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:29.165 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:51:29.184403+0000 2024-03-29T10:51:29.165 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:51:29.184403+0000 2024-03-29T10:51:29.165 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:29.165 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:29.165 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:29.165 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:51:29.166 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:51:29.166 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:29.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.183+0000 7fb3ae4b3640 1 Processor -- start 2024-03-29T10:51:29.170 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.183+0000 7fb3ae4b3640 1 -- start start 2024-03-29T10:51:29.170 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.183+0000 7fb3ae4b3640 10 monclient: init 2024-03-29T10:51:29.170 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.184+0000 7fb3ae4b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:29.170 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.184+0000 7fb3ae4b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:51:29.170 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.184+0000 7fb3ae4b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb3a80b1f60 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:29.170 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.184+0000 7fb3ae4b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb3a8071eb0 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:29.170 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.187+0000 7fb3aca50640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fb3a8143d10 0x7fb3a8146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:41648/0 (socket says 172.21.15.98:41648) 2024-03-29T10:51:29.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.187+0000 7fb3a77fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb3a8071eb0 0x7fb3a8146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:52803/0 (socket says 172.21.15.98:52803) 2024-03-29T10:51:29.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.187+0000 7fb3aca50640 1 -- 172.21.15.98:0/3093848181 learned_addr learned my addr 172.21.15.98:0/3093848181 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:29.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.187+0000 7fb3ae4b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb3a8143d10 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:29.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.187+0000 7fb3ae4b3640 1 -- 172.21.15.98:0/3093848181 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a806ef80 con 0x7fb3a80b1f60 2024-03-29T10:51:29.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.187+0000 7fb3ae4b3640 1 -- 172.21.15.98:0/3093848181 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a806f260 con 0x7fb3a8071eb0 2024-03-29T10:51:29.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.187+0000 7fb3ae4b3640 1 -- 172.21.15.98:0/3093848181 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a80b40c0 con 0x7fb3a8143d10 2024-03-29T10:51:29.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.187+0000 7fb3ae4b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:29.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.187+0000 7fb3ae4b3640 10 monclient(hunting): authenticate will time out at 1309.385724s 2024-03-29T10:51:29.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 1 -- 172.21.15.98:0/3093848181 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032210817 0 0) 0x7fb3a806f260 con 0x7fb3a8071eb0 2024-03-29T10:51:29.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:51:29.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:29.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:29.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 10 monclient(hunting): my global_id is 4468 2024-03-29T10:51:29.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 1 -- 172.21.15.98:0/3093848181 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb394003520 con 0x7fb3a8071eb0 2024-03-29T10:51:29.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 1 -- 172.21.15.98:0/3093848181 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 63776874 0 0) 0x7fb394003520 con 0x7fb3a8071eb0 2024-03-29T10:51:29.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 1 -- 172.21.15.98:0/3093848181 >> v1:172.21.15.98:6790/0 conn(0x7fb3a8143d10 legacy=0x7fb3a8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:51:29.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 1 -- 172.21.15.98:0/3093848181 >> v1:172.21.15.98:6789/0 conn(0x7fb3a80b1f60 legacy=0x7fb3a8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:51:29.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:51:29.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 1 monclient: found mon.noname-c 2024-03-29T10:51:29.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:29.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:51:29.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 1 -- 172.21.15.98:0/3093848181 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a8155d70 con 0x7fb3a8071eb0 2024-03-29T10:51:29.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:51:29.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:51:29.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:29.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.189+0000 7fb3a6ffd640 1 -- 172.21.15.98:0/3093848181 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb390001690 con 0x7fb3a8071eb0 2024-03-29T10:51:29.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.190+0000 7fb3a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:29.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.190+0000 7fb3a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:29.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.190+0000 7fb3a6ffd640 10 monclient: dump: 2024-03-29T10:51:29.174 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:29.174 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:29.174 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:29.174 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:29.175 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:29.175 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:29.175 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:29.175 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:29.175 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:29.175 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:29.176 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.191+0000 7fb3a6ffd640 1 -- 172.21.15.98:0/3093848181 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb390001920 con 0x7fb3a8071eb0 2024-03-29T10:51:29.176 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.191+0000 7fb3a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:29.176 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.191+0000 7fb3a6ffd640 1 -- 172.21.15.98:0/3093848181 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb390001cb0 con 0x7fb3a8071eb0 2024-03-29T10:51:29.176 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.191+0000 7fb3a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:29.176 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.191+0000 7fb3a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:29.176 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.191+0000 7fb3a6ffd640 10 monclient: dump: 2024-03-29T10:51:29.176 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:29.177 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:29.177 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:29.177 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:29.177 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:29.177 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:29.177 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:29.177 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:29.177 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:29.177 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:29.177 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.191+0000 7fb3ae4b3640 5 monclient: authenticate success, global_id 4468 2024-03-29T10:51:29.177 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.191+0000 7fb3ae4b3640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:29.177 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.191+0000 7fb3ae4b3640 4 set_mon_vals no callback set 2024-03-29T10:51:29.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.192+0000 7fb3ae4b3640 10 monclient: shutdown 2024-03-29T10:51:29.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.192+0000 7fb3ae4b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:29.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.192+0000 7fb3ae4b3640 1 -- 172.21.15.98:0/3093848181 >> v1:172.21.15.189:6789/0 conn(0x7fb3a8071eb0 legacy=0x7fb3a8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:29.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.192+0000 7fb3ae4b3640 1 -- 172.21.15.98:0/3093848181 shutdown_connections 2024-03-29T10:51:29.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.192+0000 7fb3ae4b3640 1 -- 172.21.15.98:0/3093848181 >> 172.21.15.98:0/3093848181 conn(0x7fb3a8066a00 msgr2=0x7fb3a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:29.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.196+0000 7fb3ae4b3640 1 -- 172.21.15.98:0/3093848181 shutdown_connections 2024-03-29T10:51:29.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.196+0000 7fb3ae4b3640 1 -- 172.21.15.98:0/3093848181 wait complete. 2024-03-29T10:51:29.180 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.196+0000 7fb3ae4b3640 10 monclient: build_initial_monmap 2024-03-29T10:51:29.180 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.196+0000 7fb3ae4b3640 10 monclient: monmap: 2024-03-29T10:51:29.180 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:29.180 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:29.180 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:51:29.180 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:51:29.180 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:29.180 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:29.180 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:29.180 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:51:29.180 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:51:29.181 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:29.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.197+0000 7fb3ae4b3640 1 librados: starting msgr at 2024-03-29T10:51:29.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.197+0000 7fb3ae4b3640 1 librados: starting objecter 2024-03-29T10:51:29.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.197+0000 7fb3ae4b3640 1 Processor -- start 2024-03-29T10:51:29.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.198+0000 7fb3ae4b3640 1 -- start start 2024-03-29T10:51:29.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.198+0000 7fb3ae4b3640 1 librados: setting wanted keys 2024-03-29T10:51:29.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.198+0000 7fb3ae4b3640 1 librados: calling monclient init 2024-03-29T10:51:29.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.198+0000 7fb3ae4b3640 10 monclient: init 2024-03-29T10:51:29.182 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.199+0000 7fb3ae4b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:29.182 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.199+0000 7fb3ae4b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:51:29.182 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.199+0000 7fb3ae4b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb3a80b1f60 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:29.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.200+0000 7fb3ae4b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb3a8143d10 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:29.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.200+0000 7fb3ae4b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb3a806fda0 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:29.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.200+0000 7fb3ae4b3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a8071770 con 0x7fb3a80b1f60 2024-03-29T10:51:29.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.200+0000 7fb3ae4b3640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a8070880 con 0x7fb3a806fda0 2024-03-29T10:51:29.184 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.200+0000 7fb3ae4b3640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a8070b60 con 0x7fb3a8143d10 2024-03-29T10:51:29.184 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.200+0000 7fb3ae4b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:29.184 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.200+0000 7fb3ae4b3640 10 monclient(hunting): authenticate will time out at 1309.398696s 2024-03-29T10:51:29.184 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.202+0000 7fb3a77fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fb3a8143d10 0x7fb3a81def60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:34606/0 (socket says 172.21.15.98:34606) 2024-03-29T10:51:29.184 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.202+0000 7fb3a77fe640 1 -- 172.21.15.98:0/4133206113 learned_addr learned my addr 172.21.15.98:0/4133206113 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:29.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.202+0000 7fb3a4ff9640 10 client.?.objecter ms_handle_connect 0x7fb3a8143d10 2024-03-29T10:51:29.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.202+0000 7fb3a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:29.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276310451 0 0) 0x7fb3a8070b60 con 0x7fb3a8143d10 2024-03-29T10:51:29.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:29.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:29.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:29.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:51:29.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb388004eb0 con 0x7fb3a8143d10 2024-03-29T10:51:29.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3901671699 0 0) 0x7fb388004eb0 con 0x7fb3a8143d10 2024-03-29T10:51:29.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3a8070b60 con 0x7fb3a8143d10 2024-03-29T10:51:29.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb390001650 con 0x7fb3a8143d10 2024-03-29T10:51:29.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:29.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:29.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:29.188 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:29.188 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:29.188 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:29.188 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:29.188 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:29.188 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:29.188 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:29.188 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:29.188 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:29.188 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:29.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:29.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 10 client.?.objecter ms_handle_connect 0x7fb3a80b1f60 2024-03-29T10:51:29.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.204+0000 7fb3a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:29.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.205+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037104289 0 0) 0x7fb3a8071770 con 0x7fb3a80b1f60 2024-03-29T10:51:29.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.205+0000 7fb3a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:29.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.205+0000 7fb3a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:29.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.205+0000 7fb3a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:29.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.205+0000 7fb3a4ff9640 10 monclient(hunting): my global_id is 4461 2024-03-29T10:51:29.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.205+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb388004eb0 con 0x7fb3a80b1f60 2024-03-29T10:51:29.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.206+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 425885663 0 0) 0x7fb388004eb0 con 0x7fb3a80b1f60 2024-03-29T10:51:29.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.206+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb388005990 con 0x7fb3a80b1f60 2024-03-29T10:51:29.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.206+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb3a0001750 con 0x7fb3a80b1f60 2024-03-29T10:51:29.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.206+0000 7fb3a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:29.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.206+0000 7fb3a4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:29.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.206+0000 7fb3a4ff9640 10 monclient(hunting): dump: 2024-03-29T10:51:29.195 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:29.195 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:29.195 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:29.195 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:29.195 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:29.195 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:29.195 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:29.195 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:29.195 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:29.195 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:29.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.206+0000 7fb3a4ff9640 10 client.?.objecter ms_handle_connect 0x7fb3a806fda0 2024-03-29T10:51:29.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.206+0000 7fb3a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:29.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.206+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992259753 0 0) 0x7fb3a8070880 con 0x7fb3a806fda0 2024-03-29T10:51:29.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:51:29.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:29.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:29.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 10 monclient(hunting): my global_id is 4471 2024-03-29T10:51:29.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb388004eb0 con 0x7fb3a806fda0 2024-03-29T10:51:29.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 973929719 0 0) 0x7fb388005990 con 0x7fb3a80b1f60 2024-03-29T10:51:29.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 >> v1:172.21.15.98:6790/0 conn(0x7fb3a8143d10 legacy=0x7fb3a81def60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:29.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 >> v1:172.21.15.189:6789/0 conn(0x7fb3a806fda0 legacy=0x7fb3a81e6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:29.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:51:29.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 1 monclient: found mon.a 2024-03-29T10:51:29.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:29.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:51:29.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a8070e40 con 0x7fb3a80b1f60 2024-03-29T10:51:29.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:51:29.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:51:29.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.207+0000 7fb3a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:29.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.208+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3a0001930 con 0x7fb3a80b1f60 2024-03-29T10:51:29.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.208+0000 7fb3a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:29.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.208+0000 7fb3ae4b3640 5 monclient: authenticate success, global_id 4461 2024-03-29T10:51:29.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.208+0000 7fb3ae4b3640 10 monclient: _renew_subs 2024-03-29T10:51:29.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.208+0000 7fb3ae4b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:51:29.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.208+0000 7fb3ae4b3640 1 -- 172.21.15.98:0/4133206113 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a8155560 con 0x7fb3a80b1f60 2024-03-29T10:51:29.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.209+0000 7fb3ae4b3640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:29.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.209+0000 7fb3ae4b3640 10 monclient: _renew_subs 2024-03-29T10:51:29.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.209+0000 7fb3ae4b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:51:29.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.209+0000 7fb3ae4b3640 1 -- 172.21.15.98:0/4133206113 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a81425a0 con 0x7fb3a80b1f60 2024-03-29T10:51:29.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.209+0000 7fb3ae4b3640 1 librados: init done 2024-03-29T10:51:29.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.209+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb3a0003790 con 0x7fb3a80b1f60 2024-03-29T10:51:29.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.209+0000 7fb3a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:29.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.209+0000 7fb3a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:29.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.209+0000 7fb3a4ff9640 10 monclient: dump: 2024-03-29T10:51:29.198 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:29.198 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:29.198 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:29.198 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:29.198 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:29.198 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:29.199 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:29.199 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:29.199 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:29.199 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:29.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.210+0000 7fb39e7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:29.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.210+0000 7fb39e7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:29.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.210+0000 7fb39e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:51:29.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.210+0000 7fb39e7fc640 1 -- 172.21.15.98:0/4133206113 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38c005080 con 0x7fb3a80b1f60 2024-03-29T10:51:29.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.211+0000 7fb3a5ffb640 4 set_mon_vals no callback set 2024-03-29T10:51:29.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.212+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fb3a0019760 con 0x7fb3a80b1f60 2024-03-29T10:51:29.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.213+0000 7fb3a77fe640 10 monclient: get_auth_request con 0x7fb38806e090 auth_method 0 2024-03-29T10:51:29.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.213+0000 7fb3a4ff9640 10 client.4461.objecter ms_handle_connect 0x7fb38806e090 2024-03-29T10:51:29.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.214+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fb3a0053270 con 0x7fb3a80b1f60 2024-03-29T10:51:29.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.214+0000 7fb3a4ff9640 10 client.4461.objecter ms_dispatch 0x7fb3a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:29.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.214+0000 7fb3a4ff9640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:29.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.214+0000 7fb3a4ff9640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:29.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.214+0000 7fb3a4ff9640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T10:51:29.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.219+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3a004f510 con 0x7fb3a80b1f60 2024-03-29T10:51:29.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.219+0000 7fb3a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:29.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.219+0000 7fb3a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:29.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.554+0000 7fb39e7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:29.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.554+0000 7fb39e7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:29.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.554+0000 7fb39e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:51:29.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.554+0000 7fb39e7fc640 1 -- 172.21.15.98:0/4133206113 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb38c005330 con 0x7fb3a80b1f60 2024-03-29T10:51:29.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.555+0000 7fb3a4ff9640 1 -- 172.21.15.98:0/4133206113 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fb3a004f6b0 con 0x7fb3a80b1f60 2024-03-29T10:51:29.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.555+0000 7fb3a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:29.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.555+0000 7fb3a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:29.538 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:51:29.539 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:29.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.559+0000 7fb39e7fc640 10 librados: watch_flush enter 2024-03-29T10:51:29.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.559+0000 7fb39e7fc640 10 librados: watch_flush exit 2024-03-29T10:51:29.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.559+0000 7fb39e7fc640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T10:51:29.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.559+0000 7fb39e7fc640 10 client.4461.objecter successfully canceled tick 2024-03-29T10:51:29.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.559+0000 7fb39e7fc640 1 -- 172.21.15.98:0/4133206113 >> v1:172.21.15.98:6816/1481799426 conn(0x7fb38806e090 legacy=0x7fb388070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:29.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.559+0000 7fb39e7fc640 10 monclient: shutdown 2024-03-29T10:51:29.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.559+0000 7fb39e7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:29.542 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.559+0000 7fb39e7fc640 1 -- 172.21.15.98:0/4133206113 >> v1:172.21.15.98:6789/0 conn(0x7fb3a80b1f60 legacy=0x7fb3a806f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:29.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.560+0000 7fb39e7fc640 1 -- 172.21.15.98:0/4133206113 shutdown_connections 2024-03-29T10:51:29.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.560+0000 7fb39e7fc640 1 -- 172.21.15.98:0/4133206113 >> 172.21.15.98:0/4133206113 conn(0x7fb3a8066a00 msgr2=0x7fb3a80b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:29.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.561+0000 7fb39e7fc640 1 -- 172.21.15.98:0/4133206113 shutdown_connections 2024-03-29T10:51:29.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.561+0000 7fb39e7fc640 1 -- 172.21.15.98:0/4133206113 wait complete. 2024-03-29T10:51:29.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:29.561+0000 7fb39e7fc640 1 librados: shutdown 2024-03-29T10:51:29.561 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:36.369 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:36.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.533+0000 7f20f9e0d640 10 monclient: get_monmap_and_config 2024-03-29T10:51:36.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.533+0000 7f20f9e0d640 10 monclient: build_initial_monmap 2024-03-29T10:51:36.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.533+0000 7f20f9e0d640 10 monclient: monmap: 2024-03-29T10:51:36.516 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:36.516 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:36.516 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:51:36.535217+0000 2024-03-29T10:51:36.516 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:51:36.535217+0000 2024-03-29T10:51:36.516 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:36.517 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:36.517 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:36.517 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:51:36.517 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:51:36.517 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:36.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.534+0000 7f20f9e0d640 1 Processor -- start 2024-03-29T10:51:36.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.534+0000 7f20f9e0d640 1 -- start start 2024-03-29T10:51:36.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.534+0000 7f20f9e0d640 10 monclient: init 2024-03-29T10:51:36.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.535+0000 7f20f9e0d640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:36.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.535+0000 7f20f9e0d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:36.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.535+0000 7f20f9e0d640 10 monclient(hunting): picked mon.noname-c con 0x7f20f40a9ba0 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:36.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.535+0000 7f20f9e0d640 10 monclient(hunting): picked mon.noname-b con 0x7f20f413b5a0 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:36.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.535+0000 7f20f9e0d640 10 monclient(hunting): picked mon.noname-a con 0x7f20f40ab550 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:36.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.535+0000 7f20f9e0d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f40ac430 con 0x7f20f40ab550 2024-03-29T10:51:36.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.535+0000 7f20f9e0d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f40ac680 con 0x7f20f40a9ba0 2024-03-29T10:51:36.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.535+0000 7f20f9e0d640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f40ac960 con 0x7f20f413b5a0 2024-03-29T10:51:36.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.535+0000 7f20f9e0d640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:36.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.535+0000 7f20f9e0d640 10 monclient(hunting): authenticate will time out at 1316.733550s 2024-03-29T10:51:36.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.538+0000 7f20f2ffd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f20f413b5a0 0x7f20f40aae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:60815/0 (socket says 172.21.15.98:60815) 2024-03-29T10:51:36.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.538+0000 7f20f2ffd640 1 -- 172.21.15.98:0/889929423 learned_addr learned my addr 172.21.15.98:0/889929423 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:36.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.541+0000 7f20f27fc640 1 -- 172.21.15.98:0/889929423 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113608311 0 0) 0x7f20f40ac960 con 0x7f20f413b5a0 2024-03-29T10:51:36.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.541+0000 7f20f27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:36.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.541+0000 7f20f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:36.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.541+0000 7f20f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:36.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.541+0000 7f20f27fc640 10 monclient(hunting): my global_id is 4505 2024-03-29T10:51:36.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.541+0000 7f20f27fc640 1 -- 172.21.15.98:0/889929423 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20dc003520 con 0x7f20f413b5a0 2024-03-29T10:51:36.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.541+0000 7f20f27fc640 1 -- 172.21.15.98:0/889929423 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3978957624 0 0) 0x7f20dc003520 con 0x7f20f413b5a0 2024-03-29T10:51:36.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.541+0000 7f20f27fc640 1 -- 172.21.15.98:0/889929423 >> v1:172.21.15.189:6789/0 conn(0x7f20f40a9ba0 legacy=0x7f20f413ae90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:36.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.542+0000 7f20f27fc640 1 -- 172.21.15.98:0/889929423 >> v1:172.21.15.98:6789/0 conn(0x7f20f40ab550 legacy=0x7f20f41498d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:36.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:36.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 1 monclient: found mon.noname-b 2024-03-29T10:51:36.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:36.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:51:36.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 1 -- 172.21.15.98:0/889929423 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f413baf0 con 0x7f20f413b5a0 2024-03-29T10:51:36.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:36.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:36.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:36.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 1 -- 172.21.15.98:0/889929423 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f20e4001690 con 0x7f20f413b5a0 2024-03-29T10:51:36.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:36.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:36.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 10 monclient: dump: 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f9e0d640 5 monclient: authenticate success, global_id 4505 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f9e0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 1 -- 172.21.15.98:0/889929423 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20e4001860 con 0x7f20f413b5a0 2024-03-29T10:51:36.530 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:36.531 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 1 -- 172.21.15.98:0/889929423 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f20e4001cb0 con 0x7f20f413b5a0 2024-03-29T10:51:36.531 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:36.531 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:36.531 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f27fc640 10 monclient: dump: 2024-03-29T10:51:36.531 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:36.531 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:36.532 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:36.532 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:36.532 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:36.532 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:36.532 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:36.532 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:36.532 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:36.532 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:36.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f9e0d640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:36.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.543+0000 7f20f9e0d640 4 set_mon_vals no callback set 2024-03-29T10:51:36.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.544+0000 7f20f9e0d640 10 monclient: shutdown 2024-03-29T10:51:36.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.544+0000 7f20f9e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:36.532 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.544+0000 7f20f9e0d640 1 -- 172.21.15.98:0/889929423 >> v1:172.21.15.98:6790/0 conn(0x7f20f413b5a0 legacy=0x7f20f40aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:36.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.544+0000 7f20f9e0d640 1 -- 172.21.15.98:0/889929423 shutdown_connections 2024-03-29T10:51:36.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.544+0000 7f20f9e0d640 1 -- 172.21.15.98:0/889929423 >> 172.21.15.98:0/889929423 conn(0x7f20f40b3a00 msgr2=0x7f20f40b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:36.533 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.546+0000 7f20f9e0d640 1 -- 172.21.15.98:0/889929423 shutdown_connections 2024-03-29T10:51:36.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.546+0000 7f20f9e0d640 1 -- 172.21.15.98:0/889929423 wait complete. 2024-03-29T10:51:36.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.547+0000 7f20f9e0d640 10 monclient: build_initial_monmap 2024-03-29T10:51:36.535 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.547+0000 7f20f9e0d640 10 monclient: monmap: 2024-03-29T10:51:36.535 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:36.535 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:36.535 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:51:36.535 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:51:36.535 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:36.535 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.548+0000 7f20f9e0d640 1 librados: starting msgr at 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.548+0000 7f20f9e0d640 1 librados: starting objecter 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.548+0000 7f20f9e0d640 1 Processor -- start 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.548+0000 7f20f9e0d640 1 -- start start 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.548+0000 7f20f9e0d640 1 librados: setting wanted keys 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.548+0000 7f20f9e0d640 1 librados: calling monclient init 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.548+0000 7f20f9e0d640 10 monclient: init 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.549+0000 7f20f9e0d640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.549+0000 7f20f9e0d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:36.536 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.549+0000 7f20f9e0d640 10 monclient(hunting): picked mon.noname-b con 0x7f20f40a9ba0 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:36.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.549+0000 7f20f9e0d640 10 monclient(hunting): picked mon.noname-c con 0x7f20f40ab550 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:36.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.549+0000 7f20f9e0d640 10 monclient(hunting): picked mon.noname-a con 0x7f20f413b5a0 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:36.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.549+0000 7f20f9e0d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f413be40 con 0x7f20f413b5a0 2024-03-29T10:51:36.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.549+0000 7f20f9e0d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f4136e20 con 0x7f20f40a9ba0 2024-03-29T10:51:36.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.549+0000 7f20f9e0d640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f41c96c0 con 0x7f20f40ab550 2024-03-29T10:51:36.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.549+0000 7f20f9e0d640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:36.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.549+0000 7f20f9e0d640 10 monclient(hunting): authenticate will time out at 1316.747428s 2024-03-29T10:51:36.537 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.552+0000 7f20f37fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f20f40a9ba0 0x7f20f41c8fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:45917/0 (socket says 172.21.15.98:45917) 2024-03-29T10:51:36.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.552+0000 7f20f37fe640 1 -- 172.21.15.98:0/230892628 learned_addr learned my addr 172.21.15.98:0/230892628 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:36.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.553+0000 7f20dbfff640 10 client.?.objecter ms_handle_connect 0x7f20f413b5a0 2024-03-29T10:51:36.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.553+0000 7f20dbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:36.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.554+0000 7f20dbfff640 10 client.?.objecter ms_handle_connect 0x7f20f40a9ba0 2024-03-29T10:51:36.538 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.554+0000 7f20dbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:36.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.555+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035859057 0 0) 0x7f20f4136e20 con 0x7f20f40a9ba0 2024-03-29T10:51:36.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.555+0000 7f20dbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:36.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.555+0000 7f20dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:36.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.555+0000 7f20dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:36.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.555+0000 7f20dbfff640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:51:36.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.555+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e0004eb0 con 0x7f20f40a9ba0 2024-03-29T10:51:36.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868122839 0 0) 0x7f20f413be40 con 0x7f20f413b5a0 2024-03-29T10:51:36.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:36.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:36.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:36.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 10 monclient(hunting): my global_id is 4464 2024-03-29T10:51:36.539 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f4136e20 con 0x7f20f413b5a0 2024-03-29T10:51:36.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 153512237 0 0) 0x7f20e0004eb0 con 0x7f20f40a9ba0 2024-03-29T10:51:36.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20f413be40 con 0x7f20f40a9ba0 2024-03-29T10:51:36.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f20ec001bf0 con 0x7f20f40a9ba0 2024-03-29T10:51:36.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:36.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:36.540 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 10 monclient(hunting): dump: 2024-03-29T10:51:36.540 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:36.540 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:36.540 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:36.540 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:36.540 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:36.540 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:36.540 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:36.541 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:36.541 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:36.541 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:36.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:36.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4108565512 0 0) 0x7f20f4136e20 con 0x7f20f413b5a0 2024-03-29T10:51:36.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20e0004eb0 con 0x7f20f413b5a0 2024-03-29T10:51:36.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f20e8001510 con 0x7f20f413b5a0 2024-03-29T10:51:36.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:36.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:36.541 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.556+0000 7f20dbfff640 10 monclient(hunting): dump: 2024-03-29T10:51:36.541 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:36.542 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:36.542 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:36.542 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:36.542 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:36.542 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:36.542 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:36.542 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:36.542 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:36.542 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:36.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20f3fff640 1 -- 172.21.15.98:0/230892628 >> v1:172.21.15.98:6789/0 conn(0x7f20f413b5a0 legacy=0x7f20f4137340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T10:51:36.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20f3fff640 1 -- 172.21.15.98:0/230892628 >> v1:172.21.15.98:6789/0 conn(0x7f20f413b5a0 legacy=0x7f20f4137340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:51:36.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20f3fff640 1 --1- 172.21.15.98:0/230892628 >> v1:172.21.15.98:6789/0 conn(0x7f20f413b5a0 0x7f20f4137340 :-1 s=OPENED pgs=710897 cs=1 l=1).handle_message read tag failed 2024-03-29T10:51:36.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20f3fff640 1 --1- 172.21.15.98:0/230892628 >> v1:172.21.15.98:6789/0 conn(0x7f20f413b5a0 0x7f20f4137340 :-1 s=OPENED pgs=710897 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:51:36.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.98:6789/0 2024-03-29T10:51:36.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 10 client.?.objecter ms_handle_connect 0x7f20f40ab550 2024-03-29T10:51:36.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:36.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1513576140 0 0) 0x7f20f413be40 con 0x7f20f40a9ba0 2024-03-29T10:51:36.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 >> v1:172.21.15.98:6790/0 conn(0x7f20f40ab550 legacy=0x7f20f41349e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:36.543 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 >> v1:172.21.15.98:6789/0 conn(0x7f20f413b5a0 legacy=0x7f20f4137340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:51:36.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:36.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 1 monclient: found mon.b 2024-03-29T10:51:36.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:36.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:36.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f41c9850 con 0x7f20f40a9ba0 2024-03-29T10:51:36.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 10 monclient: _finish_auth 0 2024-03-29T10:51:36.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:36.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:36.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20f9e0d640 5 monclient: authenticate success, global_id 4474 2024-03-29T10:51:36.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20f9e0d640 10 monclient: _renew_subs 2024-03-29T10:51:36.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20f9e0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:36.544 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20f9e0d640 1 -- 172.21.15.98:0/230892628 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f41350f0 con 0x7f20f40a9ba0 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20f9e0d640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20f9e0d640 10 monclient: _renew_subs 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20f9e0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20f9e0d640 1 -- 172.21.15.98:0/230892628 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f41c8880 con 0x7f20f40a9ba0 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20f9e0d640 1 librados: init done 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20ec002100 con 0x7f20f40a9ba0 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f20ec004e20 con 0x7f20f40a9ba0 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.557+0000 7f20dbfff640 10 monclient: dump: 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:36.545 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:36.546 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:36.546 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:36.546 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:36.546 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:36.546 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:36.546 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:36.546 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:36.546 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.560+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f20ec01adf0 con 0x7f20f40a9ba0 2024-03-29T10:51:36.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.561+0000 7f20f17fa640 4 set_mon_vals no callback set 2024-03-29T10:51:36.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.561+0000 7f20f9e0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:36.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.561+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f20ec084da0 con 0x7f20f40a9ba0 2024-03-29T10:51:36.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.561+0000 7f20dbfff640 10 client.4474.objecter ms_dispatch 0x7f20f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:36.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.561+0000 7f20dbfff640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:36.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.561+0000 7f20dbfff640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:36.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.561+0000 7f20dbfff640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T10:51:36.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.562+0000 7f20f9e0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:36.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.562+0000 7f20f9e0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:36.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.562+0000 7f20f9e0d640 1 -- 172.21.15.98:0/230892628 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f4136c50 con 0x7f20f40a9ba0 2024-03-29T10:51:36.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.562+0000 7f20f2ffd640 10 monclient: get_auth_request con 0x7f20e006de40 auth_method 0 2024-03-29T10:51:36.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.563+0000 7f20dbfff640 10 client.4474.objecter ms_handle_connect 0x7f20e006de40 2024-03-29T10:51:36.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.571+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20ec0506e0 con 0x7f20f40a9ba0 2024-03-29T10:51:36.553 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.571+0000 7f20dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:36.554 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.571+0000 7f20dbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:36.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.896+0000 7f20f9e0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:36.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.896+0000 7f20f9e0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:36.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.896+0000 7f20f9e0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:51:36.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.896+0000 7f20f9e0d640 1 -- 172.21.15.98:0/230892628 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20f404f730 con 0x7f20f40a9ba0 2024-03-29T10:51:36.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.899+0000 7f20dbfff640 1 -- 172.21.15.98:0/230892628 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f20ec084450 con 0x7f20f40a9ba0 2024-03-29T10:51:36.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.899+0000 7f20dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:36.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.899+0000 7f20dbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:36.881 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:51:36.881 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:36.884 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.902+0000 7f20f9e0d640 10 librados: watch_flush enter 2024-03-29T10:51:36.885 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.904+0000 7f20f9e0d640 10 librados: watch_flush exit 2024-03-29T10:51:36.885 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.904+0000 7f20f9e0d640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T10:51:36.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.904+0000 7f20f9e0d640 10 client.4474.objecter successfully canceled tick 2024-03-29T10:51:36.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.904+0000 7f20f9e0d640 1 -- 172.21.15.98:0/230892628 >> v1:172.21.15.98:6816/1481799426 conn(0x7f20e006de40 legacy=0x7f20e0070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:36.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.904+0000 7f20f9e0d640 10 monclient: shutdown 2024-03-29T10:51:36.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.904+0000 7f20f9e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:36.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.904+0000 7f20f9e0d640 1 -- 172.21.15.98:0/230892628 >> v1:172.21.15.189:6789/0 conn(0x7f20f40a9ba0 legacy=0x7f20f41c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:36.887 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.905+0000 7f20f9e0d640 1 -- 172.21.15.98:0/230892628 shutdown_connections 2024-03-29T10:51:36.887 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.905+0000 7f20f9e0d640 1 -- 172.21.15.98:0/230892628 >> 172.21.15.98:0/230892628 conn(0x7f20f40b3a00 msgr2=0x7f20f40b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:36.887 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.905+0000 7f20f9e0d640 1 -- 172.21.15.98:0/230892628 shutdown_connections 2024-03-29T10:51:36.887 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.905+0000 7f20f9e0d640 1 -- 172.21.15.98:0/230892628 wait complete. 2024-03-29T10:51:36.887 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:36.905+0000 7f20f9e0d640 1 librados: shutdown 2024-03-29T10:51:36.904 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:43.711 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:43.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.880+0000 7f553392b640 10 monclient: get_monmap_and_config 2024-03-29T10:51:43.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.880+0000 7f553392b640 10 monclient: build_initial_monmap 2024-03-29T10:51:43.862 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.880+0000 7f553392b640 10 monclient: monmap: 2024-03-29T10:51:43.862 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:43.862 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:43.862 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:51:43.881190+0000 2024-03-29T10:51:43.862 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:51:43.881190+0000 2024-03-29T10:51:43.862 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:43.862 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:43.863 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:43.863 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:51:43.863 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:51:43.863 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:43.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.881+0000 7f553392b640 1 Processor -- start 2024-03-29T10:51:43.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.881+0000 7f553392b640 1 -- start start 2024-03-29T10:51:43.864 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.881+0000 7f553392b640 10 monclient: init 2024-03-29T10:51:43.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.882+0000 7f553392b640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:43.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.882+0000 7f553392b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:43.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.882+0000 7f553392b640 10 monclient(hunting): picked mon.noname-c con 0x7f552c1457f0 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:43.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.882+0000 7f553392b640 10 monclient(hunting): picked mon.noname-b con 0x7f552c13f650 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:43.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.882+0000 7f553392b640 10 monclient(hunting): picked mon.noname-a con 0x7f552c0a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:43.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.882+0000 7f553392b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552c0aaef0 con 0x7f552c0a9d70 2024-03-29T10:51:43.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.882+0000 7f553392b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552c145bd0 con 0x7f552c1457f0 2024-03-29T10:51:43.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.882+0000 7f553392b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552c145eb0 con 0x7f552c13f650 2024-03-29T10:51:43.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.882+0000 7f553392b640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:43.865 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.882+0000 7f553392b640 10 monclient(hunting): authenticate will time out at 1324.079533s 2024-03-29T10:51:43.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.885+0000 7f5530ec6640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f552c13f650 0x7f552c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:55484/0 (socket says 172.21.15.98:55484) 2024-03-29T10:51:43.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.885+0000 7f5530ec6640 1 -- 172.21.15.98:0/975441773 learned_addr learned my addr 172.21.15.98:0/975441773 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:43.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.886+0000 7f5523fff640 1 -- 172.21.15.98:0/975441773 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564262081 0 0) 0x7f552c145eb0 con 0x7f552c13f650 2024-03-29T10:51:43.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.886+0000 7f5523fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:43.871 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.886+0000 7f5523fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.886+0000 7f5523fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.886+0000 7f5523fff640 10 monclient(hunting): my global_id is 4511 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.887+0000 7f5523fff640 1 -- 172.21.15.98:0/975441773 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5518003520 con 0x7f552c13f650 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.887+0000 7f5523fff640 1 -- 172.21.15.98:0/975441773 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485397945 0 0) 0x7f552c145bd0 con 0x7f552c1457f0 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.887+0000 7f5523fff640 10 monclient(hunting): _negotiate 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.887+0000 7f5523fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.887+0000 7f5523fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.887+0000 7f5523fff640 10 monclient(hunting): my global_id is 4477 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.887+0000 7f5523fff640 1 -- 172.21.15.98:0/975441773 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f552c145eb0 con 0x7f552c1457f0 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.887+0000 7f5523fff640 1 -- 172.21.15.98:0/975441773 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2632251200 0 0) 0x7f5518003520 con 0x7f552c13f650 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.887+0000 7f5523fff640 1 -- 172.21.15.98:0/975441773 >> v1:172.21.15.189:6789/0 conn(0x7f552c1457f0 legacy=0x7f552c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.887+0000 7f5523fff640 1 -- 172.21.15.98:0/975441773 >> v1:172.21.15.98:6789/0 conn(0x7f552c0a9d70 legacy=0x7f552c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 10 monclient: _finish_hunting 0 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 1 monclient: found mon.noname-b 2024-03-29T10:51:43.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 1 -- 172.21.15.98:0/975441773 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552c146340 con 0x7f552c13f650 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 10 monclient: _finish_auth 0 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 10 monclient: _check_auth_tickets 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f553392b640 5 monclient: authenticate success, global_id 4511 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f553392b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 1 -- 172.21.15.98:0/975441773 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5524001a10 con 0x7f552c13f650 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 10 monclient: dump: 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:43.873 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f553392b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 1 -- 172.21.15.98:0/975441773 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5524001c30 con 0x7f552c13f650 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f5523fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f553392b640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.888+0000 7f553392b640 4 set_mon_vals no callback set 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.889+0000 7f553392b640 10 monclient: shutdown 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.889+0000 7f553392b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:43.874 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.889+0000 7f553392b640 1 -- 172.21.15.98:0/975441773 >> v1:172.21.15.98:6790/0 conn(0x7f552c13f650 legacy=0x7f552c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:43.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.891+0000 7f553392b640 1 -- 172.21.15.98:0/975441773 shutdown_connections 2024-03-29T10:51:43.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.891+0000 7f553392b640 1 -- 172.21.15.98:0/975441773 >> 172.21.15.98:0/975441773 conn(0x7f552c066a00 msgr2=0x7f552c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:43.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.892+0000 7f553392b640 1 -- 172.21.15.98:0/975441773 shutdown_connections 2024-03-29T10:51:43.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.892+0000 7f553392b640 1 -- 172.21.15.98:0/975441773 wait complete. 2024-03-29T10:51:43.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.893+0000 7f553392b640 10 monclient: build_initial_monmap 2024-03-29T10:51:43.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.893+0000 7f553392b640 10 monclient: monmap: 2024-03-29T10:51:43.876 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:43.876 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:43.876 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.894+0000 7f553392b640 1 librados: starting msgr at 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.894+0000 7f553392b640 1 librados: starting objecter 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.894+0000 7f553392b640 1 Processor -- start 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.894+0000 7f553392b640 1 -- start start 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.894+0000 7f553392b640 1 librados: setting wanted keys 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.894+0000 7f553392b640 1 librados: calling monclient init 2024-03-29T10:51:43.877 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.894+0000 7f553392b640 10 monclient: init 2024-03-29T10:51:43.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.895+0000 7f553392b640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:43.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.895+0000 7f553392b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:51:43.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.895+0000 7f553392b640 10 monclient(hunting): picked mon.noname-b con 0x7f552c0a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:43.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.895+0000 7f553392b640 10 monclient(hunting): picked mon.noname-c con 0x7f552c13f650 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:43.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.895+0000 7f553392b640 10 monclient(hunting): picked mon.noname-a con 0x7f552c1457f0 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:43.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.895+0000 7f553392b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552c13a040 con 0x7f552c1457f0 2024-03-29T10:51:43.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.895+0000 7f553392b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552c13a7f0 con 0x7f552c0a9d70 2024-03-29T10:51:43.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.895+0000 7f553392b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552c13a980 con 0x7f552c13f650 2024-03-29T10:51:43.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.895+0000 7f553392b640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:43.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.895+0000 7f553392b640 10 monclient(hunting): authenticate will time out at 1324.092905s 2024-03-29T10:51:43.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.898+0000 7f5530ec6640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f552c13f650 0x7f552c136e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:33508/0 (socket says 172.21.15.98:33508) 2024-03-29T10:51:43.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.898+0000 7f5530ec6640 1 -- 172.21.15.98:0/1271533242 learned_addr learned my addr 172.21.15.98:0/1271533242 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:43.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.899+0000 7f5521ffb640 10 client.?.objecter ms_handle_connect 0x7f552c13f650 2024-03-29T10:51:43.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.899+0000 7f5521ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:43.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.900+0000 7f5521ffb640 10 client.?.objecter ms_handle_connect 0x7f552c0a9d70 2024-03-29T10:51:43.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.900+0000 7f5521ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:43.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.900+0000 7f5521ffb640 10 client.?.objecter ms_handle_connect 0x7f552c1457f0 2024-03-29T10:51:43.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.900+0000 7f5521ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:43.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.900+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191962444 0 0) 0x7f552c13a980 con 0x7f552c13f650 2024-03-29T10:51:43.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.900+0000 7f5521ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:43.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.900+0000 7f5521ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:43.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.901+0000 7f5521ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:43.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.901+0000 7f5521ffb640 10 monclient(hunting): my global_id is 4514 2024-03-29T10:51:43.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.901+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5510004eb0 con 0x7f552c13f650 2024-03-29T10:51:43.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.901+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255653865 0 0) 0x7f552c13a040 con 0x7f552c1457f0 2024-03-29T10:51:43.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.901+0000 7f5521ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:43.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.901+0000 7f5521ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:43.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.901+0000 7f5521ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:43.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.901+0000 7f5521ffb640 10 monclient(hunting): my global_id is 4467 2024-03-29T10:51:43.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.901+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f552c13a980 con 0x7f552c1457f0 2024-03-29T10:51:43.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.902+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975668102 0 0) 0x7f552c13a7f0 con 0x7f552c0a9d70 2024-03-29T10:51:43.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.902+0000 7f5521ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:43.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.902+0000 7f5521ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:43.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.902+0000 7f5521ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:43.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.902+0000 7f5521ffb640 10 monclient(hunting): my global_id is 4480 2024-03-29T10:51:43.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.902+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f552c13a040 con 0x7f552c0a9d70 2024-03-29T10:51:43.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 270467435 0 0) 0x7f5510004eb0 con 0x7f552c13f650 2024-03-29T10:51:43.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f552c13a7f0 con 0x7f552c13f650 2024-03-29T10:51:43.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5524004d80 con 0x7f552c13f650 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 10 monclient(hunting): dump: 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:43.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1545886554 0 0) 0x7f552c13a980 con 0x7f552c1457f0 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5510004eb0 con 0x7f552c1457f0 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 738031503 0 0) 0x7f552c13a040 con 0x7f552c0a9d70 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f552c13a980 con 0x7f552c0a9d70 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 811707471 0 0) 0x7f552c13a7f0 con 0x7f552c13f650 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 >> v1:172.21.15.189:6789/0 conn(0x7f552c0a9d70 legacy=0x7f552c139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 >> v1:172.21.15.98:6789/0 conn(0x7f552c1457f0 legacy=0x7f552c1d2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 1 monclient: found mon.c 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552c137540 con 0x7f552c13f650 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:43.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:43.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f5521ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:43.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f553392b640 5 monclient: authenticate success, global_id 4514 2024-03-29T10:51:43.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f553392b640 10 monclient: _renew_subs 2024-03-29T10:51:43.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f553392b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:43.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.903+0000 7f553392b640 1 -- 172.21.15.98:0/1271533242 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552c137790 con 0x7f552c13f650 2024-03-29T10:51:43.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.904+0000 7f553392b640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:43.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.904+0000 7f553392b640 10 monclient: _renew_subs 2024-03-29T10:51:43.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.904+0000 7f553392b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:43.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.904+0000 7f553392b640 1 -- 172.21.15.98:0/1271533242 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f552c141f90 con 0x7f552c13f650 2024-03-29T10:51:43.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.904+0000 7f553392b640 1 librados: init done 2024-03-29T10:51:43.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.904+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5524005190 con 0x7f552c13f650 2024-03-29T10:51:43.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.904+0000 7f5521ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:43.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.904+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5524005560 con 0x7f552c13f650 2024-03-29T10:51:43.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.904+0000 7f5521ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:43.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.904+0000 7f5521ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:43.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.904+0000 7f5521ffb640 10 monclient: dump: 2024-03-29T10:51:43.894 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:43.894 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:43.894 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:43.894 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:43.894 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:43.894 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:43.895 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:43.895 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:43.895 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:43.895 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:43.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.904+0000 7f55237fe640 4 set_mon_vals no callback set 2024-03-29T10:51:43.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.906+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f5524001e20 con 0x7f552c13f650 2024-03-29T10:51:43.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.907+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f5524084fb0 con 0x7f552c13f650 2024-03-29T10:51:43.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.907+0000 7f5521ffb640 10 client.4514.objecter ms_dispatch 0x7f552c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:43.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.907+0000 7f5521ffb640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:43.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.907+0000 7f5521ffb640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:43.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.907+0000 7f5521ffb640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-29T10:51:43.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.907+0000 7f55316c7640 10 monclient: get_auth_request con 0x7f551006e1d0 auth_method 0 2024-03-29T10:51:43.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.907+0000 7f553392b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:43.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.907+0000 7f553392b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:43.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.907+0000 7f553392b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:43.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.907+0000 7f553392b640 1 -- 172.21.15.98:0/1271533242 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550c005080 con 0x7f552c13f650 2024-03-29T10:51:43.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.908+0000 7f5521ffb640 10 client.4514.objecter ms_handle_connect 0x7f551006e1d0 2024-03-29T10:51:43.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.917+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5524050890 con 0x7f552c13f650 2024-03-29T10:51:43.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.917+0000 7f5521ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:43.898 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:43.917+0000 7f5521ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:44.228 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.246+0000 7f553392b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:44.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.246+0000 7f553392b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:44.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.246+0000 7f553392b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:44.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.246+0000 7f553392b640 1 -- 172.21.15.98:0/1271533242 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f550c005330 con 0x7f552c13f650 2024-03-29T10:51:44.229 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.248+0000 7f5521ffb640 1 -- 172.21.15.98:0/1271533242 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f55240527c0 con 0x7f552c13f650 2024-03-29T10:51:44.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.248+0000 7f5521ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:44.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.248+0000 7f5521ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:44.230 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:51:44.230 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:44.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.252+0000 7f553392b640 10 librados: watch_flush enter 2024-03-29T10:51:44.234 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.252+0000 7f553392b640 10 librados: watch_flush exit 2024-03-29T10:51:44.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.252+0000 7f553392b640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-29T10:51:44.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.252+0000 7f553392b640 10 client.4514.objecter successfully canceled tick 2024-03-29T10:51:44.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.252+0000 7f553392b640 1 -- 172.21.15.98:0/1271533242 >> v1:172.21.15.98:6816/1481799426 conn(0x7f551006e1d0 legacy=0x7f5510070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:44.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.252+0000 7f553392b640 10 monclient: shutdown 2024-03-29T10:51:44.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.252+0000 7f553392b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:44.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.252+0000 7f553392b640 1 -- 172.21.15.98:0/1271533242 >> v1:172.21.15.98:6790/0 conn(0x7f552c13f650 legacy=0x7f552c136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:44.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.253+0000 7f553392b640 1 -- 172.21.15.98:0/1271533242 shutdown_connections 2024-03-29T10:51:44.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.253+0000 7f553392b640 1 -- 172.21.15.98:0/1271533242 >> 172.21.15.98:0/1271533242 conn(0x7f552c066a00 msgr2=0x7f552c13e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:44.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.253+0000 7f553392b640 1 -- 172.21.15.98:0/1271533242 shutdown_connections 2024-03-29T10:51:44.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.253+0000 7f553392b640 1 -- 172.21.15.98:0/1271533242 wait complete. 2024-03-29T10:51:44.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:44.253+0000 7f553392b640 1 librados: shutdown 2024-03-29T10:51:44.252 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:51.058 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:51.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.242+0000 7f63f54d6640 10 monclient: get_monmap_and_config 2024-03-29T10:51:51.225 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.242+0000 7f63f54d6640 10 monclient: build_initial_monmap 2024-03-29T10:51:51.226 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.242+0000 7f63f54d6640 10 monclient: monmap: 2024-03-29T10:51:51.226 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:51.226 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:51.226 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:51:51.243284+0000 2024-03-29T10:51:51.226 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:51:51.243284+0000 2024-03-29T10:51:51.226 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:51.226 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:51.226 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:51.226 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:51:51.226 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:51:51.226 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:51.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.244+0000 7f63f54d6640 1 Processor -- start 2024-03-29T10:51:51.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.245+0000 7f63f54d6640 1 -- start start 2024-03-29T10:51:51.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.245+0000 7f63f54d6640 10 monclient: init 2024-03-29T10:51:51.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.246+0000 7f63f54d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:51.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.246+0000 7f63f54d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:51:51.227 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.246+0000 7f63f54d6640 10 monclient(hunting): picked mon.noname-c con 0x7f63f00accf0 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:51.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.246+0000 7f63f54d6640 10 monclient(hunting): picked mon.noname-a con 0x7f63f00ad880 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:51.230 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.249+0000 7f63f54d6640 10 monclient(hunting): picked mon.noname-b con 0x7f63f00ad0d0 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:51.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.249+0000 7f63f54d6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f00ad500 con 0x7f63f00ad880 2024-03-29T10:51:51.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.249+0000 7f63f54d6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f00aec00 con 0x7f63f00accf0 2024-03-29T10:51:51.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.249+0000 7f63f54d6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f00aeee0 con 0x7f63f00ad0d0 2024-03-29T10:51:51.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.249+0000 7f63f54d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:51.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.249+0000 7f63f54d6640 10 monclient(hunting): authenticate will time out at 1331.446492s 2024-03-29T10:51:51.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.250+0000 7f63ee7fc640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f63f00ad880 0x7f63f013d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47449/0 (socket says 172.21.15.98:47449) 2024-03-29T10:51:51.231 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.250+0000 7f63ee7fc640 1 -- 172.21.15.98:0/1630142829 learned_addr learned my addr 172.21.15.98:0/1630142829 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:51.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.252+0000 7f63edffb640 1 -- 172.21.15.98:0/1630142829 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310503643 0 0) 0x7f63f00ad500 con 0x7f63f00ad880 2024-03-29T10:51:51.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.252+0000 7f63edffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:51.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.252+0000 7f63edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:51.235 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.252+0000 7f63edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.252+0000 7f63edffb640 10 monclient(hunting): my global_id is 4470 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.252+0000 7f63edffb640 1 -- 172.21.15.98:0/1630142829 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d8003520 con 0x7f63f00ad880 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.252+0000 7f63edffb640 1 -- 172.21.15.98:0/1630142829 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468282328 0 0) 0x7f63f00aec00 con 0x7f63f00accf0 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.252+0000 7f63edffb640 10 monclient(hunting): _negotiate 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.252+0000 7f63edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.252+0000 7f63edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.252+0000 7f63edffb640 10 monclient(hunting): my global_id is 4483 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.252+0000 7f63edffb640 1 -- 172.21.15.98:0/1630142829 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f00ad500 con 0x7f63f00accf0 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 1 -- 172.21.15.98:0/1630142829 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3779366620 0 0) 0x7f63f00ad500 con 0x7f63f00accf0 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 1 -- 172.21.15.98:0/1630142829 >> v1:172.21.15.98:6790/0 conn(0x7f63f00ad0d0 legacy=0x7f63f014b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 1 -- 172.21.15.98:0/1630142829 >> v1:172.21.15.98:6789/0 conn(0x7f63f00ad880 legacy=0x7f63f013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 10 monclient: _finish_hunting 0 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 1 monclient: found mon.noname-c 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:51.236 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 1 -- 172.21.15.98:0/1630142829 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f00af1c0 con 0x7f63f00accf0 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 10 monclient: _finish_auth 0 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 10 monclient: _check_auth_tickets 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63f54d6640 5 monclient: authenticate success, global_id 4483 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 1 -- 172.21.15.98:0/1630142829 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f63e4001520 con 0x7f63f00accf0 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63f54d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 10 monclient: dump: 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:51.237 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63f54d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 1 -- 172.21.15.98:0/1630142829 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63e4001700 con 0x7f63f00accf0 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63f54d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.253+0000 7f63f54d6640 4 set_mon_vals no callback set 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.254+0000 7f63edffb640 1 -- 172.21.15.98:0/1630142829 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f63e4001a90 con 0x7f63f00accf0 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.254+0000 7f63edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.254+0000 7f63edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.254+0000 7f63edffb640 10 monclient: dump: 2024-03-29T10:51:51.238 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:51.239 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:51.239 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:51.239 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:51.239 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:51.239 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:51.239 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:51.239 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:51.239 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:51.239 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:51.240 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.254+0000 7f63f54d6640 10 monclient: shutdown 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.254+0000 7f63f54d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.254+0000 7f63f54d6640 1 -- 172.21.15.98:0/1630142829 >> v1:172.21.15.189:6789/0 conn(0x7f63f00accf0 legacy=0x7f63f00af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.256+0000 7f63f54d6640 1 -- 172.21.15.98:0/1630142829 shutdown_connections 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.256+0000 7f63f54d6640 1 -- 172.21.15.98:0/1630142829 >> 172.21.15.98:0/1630142829 conn(0x7f63f00b35e0 msgr2=0x7f63f00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.257+0000 7f63f54d6640 1 -- 172.21.15.98:0/1630142829 shutdown_connections 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.257+0000 7f63f54d6640 1 -- 172.21.15.98:0/1630142829 wait complete. 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.258+0000 7f63f54d6640 10 monclient: build_initial_monmap 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.258+0000 7f63f54d6640 10 monclient: monmap: 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:51.241 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:51.242 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:51:51.242 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:51:51.242 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:51.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.259+0000 7f63f54d6640 1 librados: starting msgr at 2024-03-29T10:51:51.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.259+0000 7f63f54d6640 1 librados: starting objecter 2024-03-29T10:51:51.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.259+0000 7f63f54d6640 1 Processor -- start 2024-03-29T10:51:51.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.259+0000 7f63f54d6640 1 -- start start 2024-03-29T10:51:51.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.259+0000 7f63f54d6640 1 librados: setting wanted keys 2024-03-29T10:51:51.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.259+0000 7f63f54d6640 1 librados: calling monclient init 2024-03-29T10:51:51.242 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.259+0000 7f63f54d6640 10 monclient: init 2024-03-29T10:51:51.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.260+0000 7f63f54d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:51.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.260+0000 7f63f54d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:51.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.260+0000 7f63f54d6640 10 monclient(hunting): picked mon.noname-c con 0x7f63f00accf0 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:51.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.260+0000 7f63f54d6640 10 monclient(hunting): picked mon.noname-b con 0x7f63f00ad0d0 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:51.243 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.260+0000 7f63f54d6640 10 monclient(hunting): picked mon.noname-a con 0x7f63f00ad880 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:51.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.260+0000 7f63f54d6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f013a900 con 0x7f63f00ad880 2024-03-29T10:51:51.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.260+0000 7f63f54d6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f01c7b30 con 0x7f63f00ad0d0 2024-03-29T10:51:51.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.260+0000 7f63f54d6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f01d3a60 con 0x7f63f00accf0 2024-03-29T10:51:51.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.260+0000 7f63f54d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:51.244 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.260+0000 7f63f54d6640 10 monclient(hunting): authenticate will time out at 1331.457567s 2024-03-29T10:51:51.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.267+0000 7f63ef7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f63f00ad880 0x7f63f01debb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47675/0 (socket says 172.21.15.98:47675) 2024-03-29T10:51:51.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.267+0000 7f63ef7fe640 1 -- 172.21.15.98:0/2044390356 learned_addr learned my addr 172.21.15.98:0/2044390356 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:51.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.271+0000 7f63d77fe640 10 client.?.objecter ms_handle_connect 0x7f63f00ad880 2024-03-29T10:51:51.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.272+0000 7f63d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:51.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.272+0000 7f63d77fe640 10 client.?.objecter ms_handle_connect 0x7f63f00ad0d0 2024-03-29T10:51:51.253 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.272+0000 7f63d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:51.256 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.273+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704644140 0 0) 0x7f63f013a900 con 0x7f63f00ad880 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.273+0000 7f63d77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.273+0000 7f63d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.273+0000 7f63d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.273+0000 7f63d77fe640 10 monclient(hunting): my global_id is 4473 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.273+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63dc004eb0 con 0x7f63f00ad880 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.273+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950455330 0 0) 0x7f63f01c7b30 con 0x7f63f00ad0d0 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.273+0000 7f63d77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.273+0000 7f63d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.273+0000 7f63d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.273+0000 7f63d77fe640 10 monclient(hunting): my global_id is 4486 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.273+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f013a900 con 0x7f63f00ad0d0 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 10 client.?.objecter ms_handle_connect 0x7f63f00accf0 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:51.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 316745241 0 0) 0x7f63dc004eb0 con 0x7f63f00ad880 2024-03-29T10:51:51.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63f01c7b30 con 0x7f63f00ad880 2024-03-29T10:51:51.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f63e00019b0 con 0x7f63f00ad880 2024-03-29T10:51:51.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:51.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:51:51.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 10 monclient(hunting): dump: 2024-03-29T10:51:51.258 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:51.258 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:51.258 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:51.258 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:51.258 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:51.258 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2333087561 0 0) 0x7f63f013a900 con 0x7f63f00ad0d0 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63dc004eb0 con 0x7f63f00ad0d0 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f63e8001b30 con 0x7f63f00ad0d0 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 10 monclient(hunting): dump: 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:51.259 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2394822320 0 0) 0x7f63f01c7b30 con 0x7f63f00ad880 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 >> v1:172.21.15.98:6790/0 conn(0x7f63f00accf0 legacy=0x7f63f01d3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 >> v1:172.21.15.189:6789/0 conn(0x7f63f00ad0d0 legacy=0x7f63f01376d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 1 monclient: found mon.a 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f01d3bf0 con 0x7f63f00ad880 2024-03-29T10:51:51.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 10 monclient: _finish_auth 0 2024-03-29T10:51:51.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:51:51.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.274+0000 7f63d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:51.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.275+0000 7f63f54d6640 5 monclient: authenticate success, global_id 4473 2024-03-29T10:51:51.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.275+0000 7f63f54d6640 10 monclient: _renew_subs 2024-03-29T10:51:51.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.275+0000 7f63f54d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:51:51.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.275+0000 7f63f54d6640 1 -- 172.21.15.98:0/2044390356 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f013ab30 con 0x7f63f00ad880 2024-03-29T10:51:51.261 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.275+0000 7f63f54d6640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:51.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.275+0000 7f63f54d6640 10 monclient: _renew_subs 2024-03-29T10:51:51.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.275+0000 7f63f54d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:51:51.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.275+0000 7f63f54d6640 1 -- 172.21.15.98:0/2044390356 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f00ac720 con 0x7f63f00ad880 2024-03-29T10:51:51.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.275+0000 7f63f54d6640 1 librados: init done 2024-03-29T10:51:51.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.275+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63e0001b10 con 0x7f63f00ad880 2024-03-29T10:51:51.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.275+0000 7f63d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:51.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.275+0000 7f63ecff9640 4 set_mon_vals no callback set 2024-03-29T10:51:51.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.277+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f63e00038c0 con 0x7f63f00ad880 2024-03-29T10:51:51.265 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.278+0000 7f63f54d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:51.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.278+0000 7f63f54d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:51.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.278+0000 7f63f54d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:51:51.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.278+0000 7f63f54d6640 1 -- 172.21.15.98:0/2044390356 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d0005080 con 0x7f63f00ad880 2024-03-29T10:51:51.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.279+0000 7f63d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:51.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.279+0000 7f63d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:51:51.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.279+0000 7f63d77fe640 10 monclient: dump: 2024-03-29T10:51:51.266 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:51.266 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:51.266 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:51.266 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:51.266 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:51.266 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:51.267 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:51.267 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:51.267 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:51.267 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:51.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.279+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f63e0003ae0 con 0x7f63f00ad880 2024-03-29T10:51:51.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.280+0000 7f63eeffd640 10 monclient: get_auth_request con 0x7f63dc06df30 auth_method 0 2024-03-29T10:51:51.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.281+0000 7f63d77fe640 10 client.4473.objecter ms_handle_connect 0x7f63dc06df30 2024-03-29T10:51:51.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.281+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f63e0084ad0 con 0x7f63f00ad880 2024-03-29T10:51:51.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.281+0000 7f63d77fe640 10 client.4473.objecter ms_dispatch 0x7f63f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:51.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.281+0000 7f63d77fe640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:51.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.281+0000 7f63d77fe640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:51.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.281+0000 7f63d77fe640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-29T10:51:51.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.293+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63e0050260 con 0x7f63f00ad880 2024-03-29T10:51:51.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.293+0000 7f63d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:51.275 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.293+0000 7f63d77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:51.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.644+0000 7f63f54d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:51.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.644+0000 7f63f54d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:51.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.644+0000 7f63f54d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:51:51.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.644+0000 7f63f54d6640 1 -- 172.21.15.98:0/2044390356 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63d0005330 con 0x7f63f00ad880 2024-03-29T10:51:51.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.647+0000 7f63d77fe640 1 -- 172.21.15.98:0/2044390356 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f63e0052360 con 0x7f63f00ad880 2024-03-29T10:51:51.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.647+0000 7f63d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:51.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.647+0000 7f63d77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:51.629 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:51:51.629 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:51.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.651+0000 7f63f54d6640 10 librados: watch_flush enter 2024-03-29T10:51:51.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.651+0000 7f63f54d6640 10 librados: watch_flush exit 2024-03-29T10:51:51.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.651+0000 7f63f54d6640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-29T10:51:51.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.651+0000 7f63f54d6640 10 client.4473.objecter successfully canceled tick 2024-03-29T10:51:51.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.651+0000 7f63f54d6640 1 -- 172.21.15.98:0/2044390356 >> v1:172.21.15.98:6816/1481799426 conn(0x7f63dc06df30 legacy=0x7f63dc0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:51.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.651+0000 7f63f54d6640 10 monclient: shutdown 2024-03-29T10:51:51.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.651+0000 7f63f54d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:51.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.651+0000 7f63f54d6640 1 -- 172.21.15.98:0/2044390356 >> v1:172.21.15.98:6789/0 conn(0x7f63f00ad880 legacy=0x7f63f01debb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:51.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.651+0000 7f63f54d6640 1 -- 172.21.15.98:0/2044390356 shutdown_connections 2024-03-29T10:51:51.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.651+0000 7f63f54d6640 1 -- 172.21.15.98:0/2044390356 >> 172.21.15.98:0/2044390356 conn(0x7f63f00b35e0 msgr2=0x7f63f00b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:51.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.652+0000 7f63f54d6640 1 -- 172.21.15.98:0/2044390356 shutdown_connections 2024-03-29T10:51:51.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.652+0000 7f63f54d6640 1 -- 172.21.15.98:0/2044390356 wait complete. 2024-03-29T10:51:51.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:51.652+0000 7f63f54d6640 1 librados: shutdown 2024-03-29T10:51:51.652 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:51:58.460 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:51:58.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.613+0000 7f3f83fed640 10 monclient: get_monmap_and_config 2024-03-29T10:51:58.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.613+0000 7f3f83fed640 10 monclient: build_initial_monmap 2024-03-29T10:51:58.595 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.613+0000 7f3f83fed640 10 monclient: monmap: 2024-03-29T10:51:58.595 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:58.596 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:58.596 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:51:58.614528+0000 2024-03-29T10:51:58.596 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:51:58.614528+0000 2024-03-29T10:51:58.596 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:58.596 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:58.596 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:58.597 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:51:58.597 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:51:58.597 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:58.597 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.614+0000 7f3f83fed640 1 Processor -- start 2024-03-29T10:51:58.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.614+0000 7f3f83fed640 1 -- start start 2024-03-29T10:51:58.602 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.614+0000 7f3f83fed640 10 monclient: init 2024-03-29T10:51:58.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.615+0000 7f3f83fed640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:58.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.615+0000 7f3f83fed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:51:58.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.615+0000 7f3f83fed640 10 monclient(hunting): picked mon.noname-b con 0x7f3f7c1457f0 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:58.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.615+0000 7f3f83fed640 10 monclient(hunting): picked mon.noname-a con 0x7f3f7c13f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:58.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.615+0000 7f3f83fed640 10 monclient(hunting): picked mon.noname-c con 0x7f3f7c0a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:58.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.615+0000 7f3f83fed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c0aaef0 con 0x7f3f7c13f650 2024-03-29T10:51:58.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.615+0000 7f3f83fed640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c145bd0 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.603 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.615+0000 7f3f83fed640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c145eb0 con 0x7f3f7c1457f0 2024-03-29T10:51:58.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.615+0000 7f3f83fed640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:58.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.615+0000 7f3f83fed640 10 monclient(hunting): authenticate will time out at 1338.812557s 2024-03-29T10:51:58.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.615+0000 7f3f81d89640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f3f7c1457f0 0x7f3f7c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:36376/0 (socket says 172.21.15.98:36376) 2024-03-29T10:51:58.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.615+0000 7f3f81d89640 1 -- 172.21.15.98:0/2114318934 learned_addr learned my addr 172.21.15.98:0/2114318934 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:58.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.616+0000 7f3f80d87640 1 -- 172.21.15.98:0/2114318934 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254207915 0 0) 0x7f3f7c145eb0 con 0x7f3f7c1457f0 2024-03-29T10:51:58.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.616+0000 7f3f80d87640 10 monclient(hunting): _negotiate 2024-03-29T10:51:58.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.616+0000 7f3f80d87640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:58.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.616+0000 7f3f80d87640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:58.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.616+0000 7f3f80d87640 10 monclient(hunting): my global_id is 4523 2024-03-29T10:51:58.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.616+0000 7f3f80d87640 1 -- 172.21.15.98:0/2114318934 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f68003520 con 0x7f3f7c1457f0 2024-03-29T10:51:58.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.616+0000 7f3f80d87640 1 -- 172.21.15.98:0/2114318934 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2144430655 0 0) 0x7f3f68003520 con 0x7f3f7c1457f0 2024-03-29T10:51:58.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.616+0000 7f3f80d87640 1 -- 172.21.15.98:0/2114318934 >> v1:172.21.15.189:6789/0 conn(0x7f3f7c0a9d70 legacy=0x7f3f7c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:58.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.616+0000 7f3f80d87640 1 -- 172.21.15.98:0/2114318934 >> v1:172.21.15.98:6789/0 conn(0x7f3f7c13f650 legacy=0x7f3f7c0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:51:58.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 10 monclient: _finish_hunting 0 2024-03-29T10:51:58.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 1 monclient: found mon.noname-b 2024-03-29T10:51:58.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:58.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:51:58.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 1 -- 172.21.15.98:0/2114318934 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c146340 con 0x7f3f7c1457f0 2024-03-29T10:51:58.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 10 monclient: _finish_auth 0 2024-03-29T10:51:58.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 10 monclient: _check_auth_tickets 2024-03-29T10:51:58.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:58.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 1 -- 172.21.15.98:0/2114318934 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3f78001690 con 0x7f3f7c1457f0 2024-03-29T10:51:58.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:58.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:51:58.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 10 monclient: dump: 2024-03-29T10:51:58.619 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:58.619 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:58.619 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:58.619 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:58.619 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:58.619 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:58.620 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:58.620 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:58.620 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:58.620 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:58.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f83fed640 5 monclient: authenticate success, global_id 4523 2024-03-29T10:51:58.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f83fed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:51:58.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 1 -- 172.21.15.98:0/2114318934 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f78001860 con 0x7f3f7c1457f0 2024-03-29T10:51:58.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:58.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 1 -- 172.21.15.98:0/2114318934 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3f78001cb0 con 0x7f3f7c1457f0 2024-03-29T10:51:58.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:58.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:58.621 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f80d87640 10 monclient: dump: 2024-03-29T10:51:58.621 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:58.621 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:58.621 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:58.621 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:58.621 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:58.621 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:58.621 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:58.622 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:58.622 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:58.622 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:58.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f83fed640 10 monclient: get_monmap_and_config success 2024-03-29T10:51:58.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.619+0000 7f3f83fed640 4 set_mon_vals no callback set 2024-03-29T10:51:58.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.620+0000 7f3f83fed640 10 monclient: shutdown 2024-03-29T10:51:58.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.620+0000 7f3f83fed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:58.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.620+0000 7f3f83fed640 1 -- 172.21.15.98:0/2114318934 >> v1:172.21.15.98:6790/0 conn(0x7f3f7c1457f0 legacy=0x7f3f7c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:58.622 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.620+0000 7f3f83fed640 1 -- 172.21.15.98:0/2114318934 shutdown_connections 2024-03-29T10:51:58.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.620+0000 7f3f83fed640 1 -- 172.21.15.98:0/2114318934 >> 172.21.15.98:0/2114318934 conn(0x7f3f7c066a00 msgr2=0x7f3f7c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:58.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.621+0000 7f3f83fed640 1 -- 172.21.15.98:0/2114318934 shutdown_connections 2024-03-29T10:51:58.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.621+0000 7f3f83fed640 1 -- 172.21.15.98:0/2114318934 wait complete. 2024-03-29T10:51:58.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.622+0000 7f3f83fed640 10 monclient: build_initial_monmap 2024-03-29T10:51:58.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.622+0000 7f3f83fed640 10 monclient: monmap: 2024-03-29T10:51:58.623 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:51:58.623 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:51:58.623 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:51:58.623 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:51:58.623 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:51:58.623 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:58.623 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:51:58.624 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:51:58.624 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:51:58.624 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:58.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.623+0000 7f3f83fed640 1 librados: starting msgr at 2024-03-29T10:51:58.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.623+0000 7f3f83fed640 1 librados: starting objecter 2024-03-29T10:51:58.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.623+0000 7f3f83fed640 1 Processor -- start 2024-03-29T10:51:58.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.623+0000 7f3f83fed640 1 -- start start 2024-03-29T10:51:58.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.623+0000 7f3f83fed640 1 librados: setting wanted keys 2024-03-29T10:51:58.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.623+0000 7f3f83fed640 1 librados: calling monclient init 2024-03-29T10:51:58.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.623+0000 7f3f83fed640 10 monclient: init 2024-03-29T10:51:58.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.624+0000 7f3f83fed640 10 monclient: _reopen_session rank -1 2024-03-29T10:51:58.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.624+0000 7f3f83fed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:51:58.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.624+0000 7f3f83fed640 10 monclient(hunting): picked mon.noname-c con 0x7f3f7c0a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T10:51:58.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.624+0000 7f3f83fed640 10 monclient(hunting): picked mon.noname-b con 0x7f3f7c13f650 addr v1:172.21.15.189:6789/0 2024-03-29T10:51:58.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.624+0000 7f3f83fed640 10 monclient(hunting): picked mon.noname-a con 0x7f3f7c1457f0 addr v1:172.21.15.98:6789/0 2024-03-29T10:51:58.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.624+0000 7f3f83fed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c1cea70 con 0x7f3f7c1457f0 2024-03-29T10:51:58.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.624+0000 7f3f83fed640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c1ced50 con 0x7f3f7c13f650 2024-03-29T10:51:58.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.624+0000 7f3f83fed640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c1cf030 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.624+0000 7f3f83fed640 10 monclient(hunting): _renew_subs 2024-03-29T10:51:58.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.624+0000 7f3f83fed640 10 monclient(hunting): authenticate will time out at 1338.821928s 2024-03-29T10:51:58.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f81d89640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f3f7c0a9d70 0x7f3f7c1cd500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:38842/0 (socket says 172.21.15.98:38842) 2024-03-29T10:51:58.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f81d89640 1 -- 172.21.15.98:0/317291090 learned_addr learned my addr 172.21.15.98:0/317291090 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:51:58.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f727fc640 10 client.?.objecter ms_handle_connect 0x7f3f7c0a9d70 2024-03-29T10:51:58.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:51:58.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952267217 0 0) 0x7f3f7c1cf030 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f727fc640 10 monclient(hunting): _negotiate 2024-03-29T10:51:58.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:51:58.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:51:58.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f727fc640 10 monclient(hunting): my global_id is 4526 2024-03-29T10:51:58.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f60004eb0 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3155469609 0 0) 0x7f3f60004eb0 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f7c1cf030 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3f78001c30 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.625+0000 7f3f727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:58.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.626+0000 7f3f727fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:51:58.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.626+0000 7f3f727fc640 10 monclient(hunting): dump: 2024-03-29T10:51:58.627 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:58.628 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:58.628 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:58.628 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:58.628 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:58.628 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:58.628 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:58.628 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:58.628 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:58.628 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:58.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.626+0000 7f3f727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:51:58.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.626+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2713075871 0 0) 0x7f3f7c1cf030 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.626+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 >> v1:172.21.15.189:6789/0 conn(0x7f3f7c13f650 legacy=0x7f3f7c1cb7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:51:58.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.626+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 >> v1:172.21.15.98:6789/0 conn(0x7f3f7c1457f0 legacy=0x7f3f7c1e2d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:51:58.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 10 monclient: _finish_hunting 0 2024-03-29T10:51:58.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 1 monclient: found mon.c 2024-03-29T10:51:58.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:51:58.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:58.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c1ccf60 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 10 monclient: _finish_auth 0 2024-03-29T10:51:58.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 10 monclient: _check_auth_tickets 2024-03-29T10:51:58.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:51:58.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f83fed640 5 monclient: authenticate success, global_id 4526 2024-03-29T10:51:58.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f83fed640 10 monclient: _renew_subs 2024-03-29T10:51:58.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f83fed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:58.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f83fed640 1 -- 172.21.15.98:0/317291090 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7c1cb340 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f78004a90 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:51:58.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f83fed640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:51:58.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f83fed640 10 monclient: _renew_subs 2024-03-29T10:51:58.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3f78004ee0 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f83fed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:58.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f83fed640 1 -- 172.21.15.98:0/317291090 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7c1ccbb0 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f73fff640 4 set_mon_vals no callback set 2024-03-29T10:51:58.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f83fed640 1 librados: init done 2024-03-29T10:51:58.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:51:58.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:51:58.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.627+0000 7f3f727fc640 10 monclient: dump: 2024-03-29T10:51:58.631 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:51:58.631 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:51:58.631 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:58.631 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:51:58.631 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:51:58.631 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:51:58.631 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:51:58.631 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:51:58.631 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:51:58.632 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:51:58.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.628+0000 7f3f83fed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:58.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.628+0000 7f3f83fed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:58.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.628+0000 7f3f83fed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:58.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.628+0000 7f3f83fed640 1 -- 172.21.15.98:0/317291090 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7c1cd1a0 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.631+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f3f78005040 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.631+0000 7f3f81588640 10 monclient: get_auth_request con 0x7f3f6006dc80 auth_method 0 2024-03-29T10:51:58.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.631+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f3f78084a00 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.631+0000 7f3f727fc640 10 client.4526.objecter ms_dispatch 0x7f3f7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:51:58.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.631+0000 7f3f727fc640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:51:58.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.631+0000 7f3f727fc640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:51:58.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.632+0000 7f3f727fc640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T10:51:58.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.632+0000 7f3f727fc640 10 client.4526.objecter ms_handle_connect 0x7f3f6006dc80 2024-03-29T10:51:58.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.637+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f780502e0 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.637+0000 7f3f727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:51:58.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.637+0000 7f3f727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:51:58.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.966+0000 7f3f83fed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:58.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.966+0000 7f3f83fed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:58.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.966+0000 7f3f83fed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:51:58.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.966+0000 7f3f83fed640 1 -- 172.21.15.98:0/317291090 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f7c075f10 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.967+0000 7f3f727fc640 1 -- 172.21.15.98:0/317291090 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f3f78052210 con 0x7f3f7c0a9d70 2024-03-29T10:51:58.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.968+0000 7f3f727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:51:58.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.968+0000 7f3f727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:51:58.950 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:51:58.950 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:51:58.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.971+0000 7f3f83fed640 10 librados: watch_flush enter 2024-03-29T10:51:58.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.971+0000 7f3f83fed640 10 librados: watch_flush exit 2024-03-29T10:51:58.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.972+0000 7f3f83fed640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T10:51:58.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.972+0000 7f3f83fed640 10 client.4526.objecter successfully canceled tick 2024-03-29T10:51:58.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.973+0000 7f3f83fed640 1 -- 172.21.15.98:0/317291090 >> v1:172.21.15.98:6816/1481799426 conn(0x7f3f6006dc80 legacy=0x7f3f60070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:58.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.973+0000 7f3f83fed640 10 monclient: shutdown 2024-03-29T10:51:58.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.973+0000 7f3f83fed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:51:58.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.973+0000 7f3f83fed640 1 -- 172.21.15.98:0/317291090 >> v1:172.21.15.98:6790/0 conn(0x7f3f7c0a9d70 legacy=0x7f3f7c1cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:51:58.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.973+0000 7f3f83fed640 1 -- 172.21.15.98:0/317291090 shutdown_connections 2024-03-29T10:51:58.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.973+0000 7f3f83fed640 1 -- 172.21.15.98:0/317291090 >> 172.21.15.98:0/317291090 conn(0x7f3f7c066a00 msgr2=0x7f3f7c0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:51:58.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.973+0000 7f3f83fed640 1 -- 172.21.15.98:0/317291090 shutdown_connections 2024-03-29T10:51:58.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.973+0000 7f3f83fed640 1 -- 172.21.15.98:0/317291090 wait complete. 2024-03-29T10:51:58.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:51:58.974+0000 7f3f83fed640 1 librados: shutdown 2024-03-29T10:51:58.973 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:05.783 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:05.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.951+0000 7f804b110640 10 monclient: get_monmap_and_config 2024-03-29T10:52:05.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.951+0000 7f804b110640 10 monclient: build_initial_monmap 2024-03-29T10:52:05.933 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.951+0000 7f804b110640 10 monclient: monmap: 2024-03-29T10:52:05.933 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:05.933 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:05.933 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:52:05.952335+0000 2024-03-29T10:52:05.933 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:52:05.952335+0000 2024-03-29T10:52:05.933 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:05.933 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:05.933 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:05.934 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:52:05.934 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:52:05.934 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:05.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.952+0000 7f804b110640 1 Processor -- start 2024-03-29T10:52:05.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.952+0000 7f804b110640 1 -- start start 2024-03-29T10:52:05.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.952+0000 7f804b110640 10 monclient: init 2024-03-29T10:52:05.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.953+0000 7f804b110640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:05.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.953+0000 7f804b110640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:05.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.953+0000 7f804b110640 10 monclient(hunting): picked mon.noname-a con 0x7f80441422b0 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:05.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.953+0000 7f804b110640 10 monclient(hunting): picked mon.noname-c con 0x7f8044142690 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:05.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.953+0000 7f804b110640 10 monclient(hunting): picked mon.noname-b con 0x7f80440b17d0 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:05.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.953+0000 7f8048eac640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f80441422b0 0x7f804413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:59038/0 (socket says 172.21.15.98:59038) 2024-03-29T10:52:05.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.953+0000 7f8048eac640 1 -- 172.21.15.98:0/3423217119 learned_addr learned my addr 172.21.15.98:0/3423217119 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:05.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.953+0000 7f804b110640 1 -- 172.21.15.98:0/3423217119 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8044142c80 con 0x7f80441422b0 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.953+0000 7f804b110640 1 -- 172.21.15.98:0/3423217119 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80440aae40 con 0x7f8044142690 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.953+0000 7f804b110640 1 -- 172.21.15.98:0/3423217119 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804413fbd0 con 0x7f80440b17d0 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.953+0000 7f804b110640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.953+0000 7f804b110640 10 monclient(hunting): authenticate will time out at 1346.150862s 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 1 -- 172.21.15.98:0/3423217119 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294978517 0 0) 0x7f804413fbd0 con 0x7f80440b17d0 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient(hunting): my global_id is 4529 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 1 -- 172.21.15.98:0/3423217119 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8034003520 con 0x7f80440b17d0 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 1 -- 172.21.15.98:0/3423217119 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847818128 0 0) 0x7f8044142c80 con 0x7f80441422b0 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:05.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient(hunting): my global_id is 4476 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 1 -- 172.21.15.98:0/3423217119 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f804413fbd0 con 0x7f80441422b0 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 1 -- 172.21.15.98:0/3423217119 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634298637 0 0) 0x7f80440aae40 con 0x7f8044142690 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient(hunting): my global_id is 4492 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 1 -- 172.21.15.98:0/3423217119 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8044142c80 con 0x7f8044142690 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 1 -- 172.21.15.98:0/3423217119 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1654846846 0 0) 0x7f8034003520 con 0x7f80440b17d0 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 1 -- 172.21.15.98:0/3423217119 >> v1:172.21.15.189:6789/0 conn(0x7f8044142690 legacy=0x7f80440b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 1 -- 172.21.15.98:0/3423217119 >> v1:172.21.15.98:6789/0 conn(0x7f80441422b0 legacy=0x7f804413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 1 monclient: found mon.noname-b 2024-03-29T10:52:05.950 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 1 -- 172.21.15.98:0/3423217119 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8044151940 con 0x7f80440b17d0 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient: _finish_auth 0 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f804b110640 5 monclient: authenticate success, global_id 4529 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f804b110640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 1 -- 172.21.15.98:0/3423217119 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f803c001940 con 0x7f80440b17d0 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f80437fe640 10 monclient: dump: 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:05.951 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.954+0000 7f804b110640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.955+0000 7f80437fe640 1 -- 172.21.15.98:0/3423217119 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f803c001b60 con 0x7f80440b17d0 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.955+0000 7f80437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.955+0000 7f804b110640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.955+0000 7f804b110640 4 set_mon_vals no callback set 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.955+0000 7f80437fe640 1 -- 172.21.15.98:0/3423217119 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f803c001e70 con 0x7f80440b17d0 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.955+0000 7f80437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.955+0000 7f80437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:05.952 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.955+0000 7f80437fe640 10 monclient: dump: 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.956+0000 7f804b110640 10 monclient: shutdown 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.956+0000 7f804b110640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.956+0000 7f804b110640 1 -- 172.21.15.98:0/3423217119 >> v1:172.21.15.98:6790/0 conn(0x7f80440b17d0 legacy=0x7f80440b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:05.953 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.956+0000 7f804b110640 1 -- 172.21.15.98:0/3423217119 shutdown_connections 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.956+0000 7f804b110640 1 -- 172.21.15.98:0/3423217119 >> 172.21.15.98:0/3423217119 conn(0x7f8044066a00 msgr2=0x7f8044066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.957+0000 7f804b110640 1 -- 172.21.15.98:0/3423217119 shutdown_connections 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.957+0000 7f804b110640 1 -- 172.21.15.98:0/3423217119 wait complete. 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.958+0000 7f804b110640 10 monclient: build_initial_monmap 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.958+0000 7f804b110640 10 monclient: monmap: 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:52:05.954 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.958+0000 7f804b110640 1 librados: starting msgr at 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.958+0000 7f804b110640 1 librados: starting objecter 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.958+0000 7f804b110640 1 Processor -- start 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.958+0000 7f804b110640 1 -- start start 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.958+0000 7f804b110640 1 librados: setting wanted keys 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.958+0000 7f804b110640 1 librados: calling monclient init 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.958+0000 7f804b110640 10 monclient: init 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f804b110640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f804b110640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f804b110640 10 monclient(hunting): picked mon.noname-b con 0x7f80440b17d0 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f804b110640 10 monclient(hunting): picked mon.noname-c con 0x7f80441422b0 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f8043fff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f80441422b0 0x7f80440b5880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:44516/0 (socket says 172.21.15.98:44516) 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f8043fff640 1 -- 172.21.15.98:0/3890099133 learned_addr learned my addr 172.21.15.98:0/3890099133 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f804b110640 10 monclient(hunting): picked mon.noname-a con 0x7f8044142690 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:05.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f804b110640 1 -- 172.21.15.98:0/3890099133 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804406ebc0 con 0x7f8044142690 2024-03-29T10:52:05.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f804b110640 1 -- 172.21.15.98:0/3890099133 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804406eea0 con 0x7f80440b17d0 2024-03-29T10:52:05.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f804b110640 1 -- 172.21.15.98:0/3890099133 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804406f180 con 0x7f80441422b0 2024-03-29T10:52:05.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f804b110640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:05.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f804b110640 10 monclient(hunting): authenticate will time out at 1346.156756s 2024-03-29T10:52:05.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f80417fa640 10 client.?.objecter ms_handle_connect 0x7f80441422b0 2024-03-29T10:52:05.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.959+0000 7f80417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:05.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.960+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846263186 0 0) 0x7f804406f180 con 0x7f80441422b0 2024-03-29T10:52:05.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.960+0000 7f80417fa640 10 monclient(hunting): _negotiate 2024-03-29T10:52:05.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.960+0000 7f80417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:05.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.960+0000 7f80417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:05.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.960+0000 7f80417fa640 10 monclient(hunting): my global_id is 4532 2024-03-29T10:52:05.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.960+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8028004eb0 con 0x7f80441422b0 2024-03-29T10:52:05.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.960+0000 7f80417fa640 10 client.?.objecter ms_handle_connect 0x7f8044142690 2024-03-29T10:52:05.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.960+0000 7f80417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:05.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 10 client.?.objecter ms_handle_connect 0x7f80440b17d0 2024-03-29T10:52:05.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:05.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2560827592 0 0) 0x7f8028004eb0 con 0x7f80441422b0 2024-03-29T10:52:05.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f804406f180 con 0x7f80441422b0 2024-03-29T10:52:05.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8030001970 con 0x7f80441422b0 2024-03-29T10:52:05.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:05.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:05.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 10 monclient(hunting): dump: 2024-03-29T10:52:05.957 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:05.957 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:05.958 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:05.958 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:05.958 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:05.958 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:05.958 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:05.958 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:05.958 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:05.958 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:05.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:05.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569468856 0 0) 0x7f804406ebc0 con 0x7f8044142690 2024-03-29T10:52:05.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 10 monclient(hunting): _negotiate 2024-03-29T10:52:05.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:05.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:05.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 10 monclient(hunting): my global_id is 4479 2024-03-29T10:52:05.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.961+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8028004eb0 con 0x7f8044142690 2024-03-29T10:52:05.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 983341237 0 0) 0x7f804406f180 con 0x7f80441422b0 2024-03-29T10:52:05.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 >> v1:172.21.15.189:6789/0 conn(0x7f80440b17d0 legacy=0x7f804412bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:05.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 >> v1:172.21.15.98:6789/0 conn(0x7f8044142690 legacy=0x7f80440b5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:05.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f80417fa640 10 monclient: _finish_hunting 0 2024-03-29T10:52:05.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f80417fa640 1 monclient: found mon.c 2024-03-29T10:52:05.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f80417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:05.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f80417fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:05.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804406f460 con 0x7f80441422b0 2024-03-29T10:52:05.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f80417fa640 10 monclient: _finish_auth 0 2024-03-29T10:52:05.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f80417fa640 10 monclient: _check_auth_tickets 2024-03-29T10:52:05.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f80417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:05.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f804b110640 5 monclient: authenticate success, global_id 4532 2024-03-29T10:52:05.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f804b110640 10 monclient: _renew_subs 2024-03-29T10:52:05.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f804b110640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:05.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f804b110640 1 -- 172.21.15.98:0/3890099133 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f804406e720 con 0x7f80441422b0 2024-03-29T10:52:05.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f804b110640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:05.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f804b110640 10 monclient: _renew_subs 2024-03-29T10:52:05.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f804b110640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:05.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f804b110640 1 -- 172.21.15.98:0/3890099133 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f804412c500 con 0x7f80441422b0 2024-03-29T10:52:05.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f804b110640 1 librados: init done 2024-03-29T10:52:05.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.962+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8030001d50 con 0x7f80441422b0 2024-03-29T10:52:05.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.963+0000 7f80417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:05.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.963+0000 7f8042ffd640 4 set_mon_vals no callback set 2024-03-29T10:52:05.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.963+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f8030003a60 con 0x7f80441422b0 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.963+0000 7f80417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.963+0000 7f80417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.963+0000 7f80417fa640 10 monclient: dump: 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.964+0000 7f804b110640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:05.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.964+0000 7f804b110640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:05.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.964+0000 7f804b110640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:05.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.964+0000 7f804b110640 1 -- 172.21.15.98:0/3890099133 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8024005080 con 0x7f80441422b0 2024-03-29T10:52:05.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.964+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f8030003c80 con 0x7f80441422b0 2024-03-29T10:52:05.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.965+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f8030084b20 con 0x7f80441422b0 2024-03-29T10:52:05.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.965+0000 7f80417fa640 10 client.4532.objecter ms_dispatch 0x7f8044000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:05.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.965+0000 7f80417fa640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:05.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.965+0000 7f80417fa640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:05.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.965+0000 7f80417fa640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T10:52:05.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.965+0000 7f8048eac640 10 monclient: get_auth_request con 0x7f802806dd60 auth_method 0 2024-03-29T10:52:05.962 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.966+0000 7f80417fa640 10 client.4532.objecter ms_handle_connect 0x7f802806dd60 2024-03-29T10:52:05.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.976+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8030050400 con 0x7f80441422b0 2024-03-29T10:52:05.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.976+0000 7f80417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:05.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:05.976+0000 7f80417fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:06.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.334+0000 7f804b110640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:06.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.334+0000 7f804b110640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:06.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.334+0000 7f804b110640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:06.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.334+0000 7f804b110640 1 -- 172.21.15.98:0/3890099133 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8024005330 con 0x7f80441422b0 2024-03-29T10:52:06.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.335+0000 7f80417fa640 1 -- 172.21.15.98:0/3890099133 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f8030052330 con 0x7f80441422b0 2024-03-29T10:52:06.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.335+0000 7f80417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:06.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.335+0000 7f80417fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:06.317 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:52:06.317 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:06.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.338+0000 7f804b110640 10 librados: watch_flush enter 2024-03-29T10:52:06.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.338+0000 7f804b110640 10 librados: watch_flush exit 2024-03-29T10:52:06.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.338+0000 7f804b110640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T10:52:06.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.338+0000 7f804b110640 10 client.4532.objecter successfully canceled tick 2024-03-29T10:52:06.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.338+0000 7f804b110640 1 -- 172.21.15.98:0/3890099133 >> v1:172.21.15.98:6816/1481799426 conn(0x7f802806dd60 legacy=0x7f8028070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:06.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.338+0000 7f804b110640 10 monclient: shutdown 2024-03-29T10:52:06.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.338+0000 7f804b110640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:06.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.338+0000 7f804b110640 1 -- 172.21.15.98:0/3890099133 >> v1:172.21.15.98:6790/0 conn(0x7f80441422b0 legacy=0x7f80440b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:06.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.339+0000 7f804b110640 1 -- 172.21.15.98:0/3890099133 shutdown_connections 2024-03-29T10:52:06.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.339+0000 7f804b110640 1 -- 172.21.15.98:0/3890099133 >> 172.21.15.98:0/3890099133 conn(0x7f8044066a00 msgr2=0x7f804413e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:06.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.339+0000 7f804b110640 1 -- 172.21.15.98:0/3890099133 shutdown_connections 2024-03-29T10:52:06.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.339+0000 7f804b110640 1 -- 172.21.15.98:0/3890099133 wait complete. 2024-03-29T10:52:06.321 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:06.339+0000 7f804b110640 1 librados: shutdown 2024-03-29T10:52:06.336 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:13.143 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:13.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.321+0000 7fa159806640 10 monclient: get_monmap_and_config 2024-03-29T10:52:13.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.321+0000 7fa159806640 10 monclient: build_initial_monmap 2024-03-29T10:52:13.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.321+0000 7fa159806640 10 monclient: monmap: 2024-03-29T10:52:13.632 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:13.632 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:13.632 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:52:13.322405+0000 2024-03-29T10:52:13.632 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:52:13.322405+0000 2024-03-29T10:52:13.632 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:13.632 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:13.632 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:13.632 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:52:13.632 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:52:13.633 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:13.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.322+0000 7fa159806640 1 Processor -- start 2024-03-29T10:52:13.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.322+0000 7fa159806640 1 -- start start 2024-03-29T10:52:13.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.322+0000 7fa159806640 10 monclient: init 2024-03-29T10:52:13.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.323+0000 7fa159806640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:13.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.323+0000 7fa159806640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:13.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.324+0000 7fa159806640 10 monclient(hunting): picked mon.noname-a con 0x7fa14c0068c0 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:13.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.324+0000 7fa159806640 10 monclient(hunting): picked mon.noname-b con 0x7fa14c006ca0 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:13.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.324+0000 7fa159806640 10 monclient(hunting): picked mon.noname-c con 0x7fa14c02b4f0 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:13.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.324+0000 7fa159806640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c02c860 con 0x7fa14c0068c0 2024-03-29T10:52:13.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.324+0000 7fa159806640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c02afd0 con 0x7fa14c02b4f0 2024-03-29T10:52:13.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.324+0000 7fa159806640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c02abd0 con 0x7fa14c006ca0 2024-03-29T10:52:13.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.324+0000 7fa159806640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:13.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.324+0000 7fa159806640 10 monclient(hunting): authenticate will time out at 1353.521319s 2024-03-29T10:52:13.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.325+0000 7fa152ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa14c0068c0 0x7fa14c0c0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54780/0 (socket says 172.21.15.98:54780) 2024-03-29T10:52:13.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.325+0000 7fa152ffd640 1 -- 172.21.15.98:0/1363307559 learned_addr learned my addr 172.21.15.98:0/1363307559 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:13.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.325+0000 7fa151ffb640 1 -- 172.21.15.98:0/1363307559 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830336427 0 0) 0x7fa14c02c860 con 0x7fa14c0068c0 2024-03-29T10:52:13.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.325+0000 7fa151ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:13.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.325+0000 7fa151ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:13.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.325+0000 7fa151ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:13.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.325+0000 7fa151ffb640 10 monclient(hunting): my global_id is 4482 2024-03-29T10:52:13.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.325+0000 7fa151ffb640 1 -- 172.21.15.98:0/1363307559 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa140003520 con 0x7fa14c0068c0 2024-03-29T10:52:13.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.326+0000 7fa151ffb640 1 -- 172.21.15.98:0/1363307559 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2271835960 0 0) 0x7fa140003520 con 0x7fa14c0068c0 2024-03-29T10:52:13.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.326+0000 7fa151ffb640 1 -- 172.21.15.98:0/1363307559 >> v1:172.21.15.98:6790/0 conn(0x7fa14c006ca0 legacy=0x7fa14c0c6d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:13.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa151ffb640 1 -- 172.21.15.98:0/1363307559 >> v1:172.21.15.189:6789/0 conn(0x7fa14c02b4f0 legacy=0x7fa14c0cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:13.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa151ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:13.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa151ffb640 1 monclient: found mon.noname-a 2024-03-29T10:52:13.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa151ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa151ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa151ffb640 1 -- 172.21.15.98:0/1363307559 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c02ad60 con 0x7fa14c0068c0 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa151ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa151ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa151ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa159806640 5 monclient: authenticate success, global_id 4482 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa159806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa151ffb640 1 -- 172.21.15.98:0/1363307559 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa144001690 con 0x7fa14c0068c0 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa151ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa151ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa151ffb640 10 monclient: dump: 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:13.643 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.328+0000 7fa159806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.329+0000 7fa151ffb640 1 -- 172.21.15.98:0/1363307559 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1440017f0 con 0x7fa14c0068c0 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.329+0000 7fa151ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.329+0000 7fa159806640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.329+0000 7fa159806640 4 set_mon_vals no callback set 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.330+0000 7fa159806640 10 monclient: shutdown 2024-03-29T10:52:13.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.330+0000 7fa159806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:13.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.330+0000 7fa159806640 1 -- 172.21.15.98:0/1363307559 >> v1:172.21.15.98:6789/0 conn(0x7fa14c0068c0 legacy=0x7fa14c0c0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:13.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.330+0000 7fa159806640 1 -- 172.21.15.98:0/1363307559 shutdown_connections 2024-03-29T10:52:13.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.330+0000 7fa159806640 1 -- 172.21.15.98:0/1363307559 >> 172.21.15.98:0/1363307559 conn(0x7fa14c005920 msgr2=0x7fa14c005d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:13.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.334+0000 7fa159806640 1 -- 172.21.15.98:0/1363307559 shutdown_connections 2024-03-29T10:52:13.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.336+0000 7fa159806640 1 -- 172.21.15.98:0/1363307559 wait complete. 2024-03-29T10:52:13.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.337+0000 7fa159806640 10 monclient: build_initial_monmap 2024-03-29T10:52:13.645 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.337+0000 7fa159806640 10 monclient: monmap: 2024-03-29T10:52:13.645 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:13.645 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:13.645 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.337+0000 7fa159806640 1 librados: starting msgr at 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.337+0000 7fa159806640 1 librados: starting objecter 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.337+0000 7fa159806640 1 Processor -- start 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.338+0000 7fa159806640 1 -- start start 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.338+0000 7fa159806640 1 librados: setting wanted keys 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.338+0000 7fa159806640 1 librados: calling monclient init 2024-03-29T10:52:13.646 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.338+0000 7fa159806640 10 monclient: init 2024-03-29T10:52:13.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.339+0000 7fa159806640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:13.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.339+0000 7fa159806640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:13.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.339+0000 7fa159806640 10 monclient(hunting): picked mon.noname-b con 0x7fa14c02b4f0 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:13.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.339+0000 7fa159806640 10 monclient(hunting): picked mon.noname-a con 0x7fa14c02e740 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:13.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.339+0000 7fa159806640 10 monclient(hunting): picked mon.noname-c con 0x7fa14c0c4040 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:13.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.339+0000 7fa159806640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c02db80 con 0x7fa14c02e740 2024-03-29T10:52:13.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.339+0000 7fa159806640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c0d6440 con 0x7fa14c02b4f0 2024-03-29T10:52:13.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.339+0000 7fa159806640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c0d61c0 con 0x7fa14c0c4040 2024-03-29T10:52:13.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.339+0000 7fa159806640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:13.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.339+0000 7fa159806640 10 monclient(hunting): authenticate will time out at 1353.536801s 2024-03-29T10:52:13.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.344+0000 7fa1537fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fa14c0c4040 0x7fa14c0d5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:50205/0 (socket says 172.21.15.98:50205) 2024-03-29T10:52:13.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.344+0000 7fa1537fe640 1 -- 172.21.15.98:0/2343158689 learned_addr learned my addr 172.21.15.98:0/2343158689 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:13.647 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.344+0000 7fa14bfff640 10 client.?.objecter ms_handle_connect 0x7fa14c02e740 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.344+0000 7fa14bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.344+0000 7fa14bfff640 10 client.?.objecter ms_handle_connect 0x7fa14c02b4f0 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.344+0000 7fa14bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.345+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196602069 0 0) 0x7fa14c0d6440 con 0x7fa14c02b4f0 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.345+0000 7fa14bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.345+0000 7fa14bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.345+0000 7fa14bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.345+0000 7fa14bfff640 10 monclient(hunting): my global_id is 4501 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.345+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa134004eb0 con 0x7fa14c02b4f0 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.345+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376496953 0 0) 0x7fa14c02db80 con 0x7fa14c02e740 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.345+0000 7fa14bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:13.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 10 monclient(hunting): my global_id is 4485 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa14c0d6440 con 0x7fa14c02e740 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1001695185 0 0) 0x7fa134004eb0 con 0x7fa14c02b4f0 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa14c02db80 con 0x7fa14c02b4f0 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa144001810 con 0x7fa14c02b4f0 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 10 monclient(hunting): dump: 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:13.649 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:13.650 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:13.650 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:13.650 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:13.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:13.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3021104650 0 0) 0x7fa14c0d6440 con 0x7fa14c02e740 2024-03-29T10:52:13.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa134004eb0 con 0x7fa14c02e740 2024-03-29T10:52:13.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa13c0017d0 con 0x7fa14c02e740 2024-03-29T10:52:13.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:13.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:13.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 10 monclient(hunting): dump: 2024-03-29T10:52:13.651 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:13.651 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:13.651 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3836808520 0 0) 0x7fa14c02db80 con 0x7fa14c02b4f0 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 >> v1:172.21.15.98:6790/0 conn(0x7fa14c0c4040 legacy=0x7fa14c0d5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 >> v1:172.21.15.98:6789/0 conn(0x7fa14c02e740 legacy=0x7fa14c0d53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 1 monclient: found mon.b 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:13.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c02f7a0 con 0x7fa14c02b4f0 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.346+0000 7fa14bfff640 10 monclient: _finish_auth 0 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.347+0000 7fa14bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.347+0000 7fa14bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.347+0000 7fa159806640 5 monclient: authenticate success, global_id 4501 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.347+0000 7fa159806640 10 monclient: _renew_subs 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.347+0000 7fa159806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.347+0000 7fa159806640 1 -- 172.21.15.98:0/2343158689 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14c02fa80 con 0x7fa14c02b4f0 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.347+0000 7fa159806640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.347+0000 7fa159806640 10 monclient: _renew_subs 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.347+0000 7fa159806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.347+0000 7fa159806640 1 -- 172.21.15.98:0/2343158689 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14c02d420 con 0x7fa14c02b4f0 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.347+0000 7fa159806640 1 librados: init done 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.348+0000 7fa159806640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.348+0000 7fa159806640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:13.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.348+0000 7fa159806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.348+0000 7fa159806640 1 -- 172.21.15.98:0/2343158689 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14c004d00 con 0x7fa14c02b4f0 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.348+0000 7fa14bfff640 10 client.4501.objecter ms_handle_connect 0x7fa14c0c4040 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.348+0000 7fa14bfff640 10 client.4501.objecter resend_mon_ops 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.348+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa144001a60 con 0x7fa14c02b4f0 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.348+0000 7fa14bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.348+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa144004dd0 con 0x7fa14c02b4f0 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.348+0000 7fa14bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.348+0000 7fa14bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.348+0000 7fa14bfff640 10 monclient: dump: 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:13.654 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.350+0000 7fa1517fa640 4 set_mon_vals no callback set 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.351+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fa144004f30 con 0x7fa14c02b4f0 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.352+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fa144084aa0 con 0x7fa14c02b4f0 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.352+0000 7fa14bfff640 10 client.4501.objecter ms_dispatch 0x7fa154000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.352+0000 7fa1527fc640 10 monclient: get_auth_request con 0x7fa13406ded0 auth_method 0 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.352+0000 7fa14bfff640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.352+0000 7fa14bfff640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.352+0000 7fa14bfff640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.353+0000 7fa14bfff640 10 client.4501.objecter ms_handle_connect 0x7fa13406ded0 2024-03-29T10:52:13.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.357+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa144050380 con 0x7fa14c02b4f0 2024-03-29T10:52:13.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.357+0000 7fa14bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:13.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.357+0000 7fa14bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:13.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.720+0000 7fa159806640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:13.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.720+0000 7fa159806640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:13.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.720+0000 7fa159806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:52:13.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.720+0000 7fa159806640 1 -- 172.21.15.98:0/2343158689 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa14c0c78b0 con 0x7fa14c02b4f0 2024-03-29T10:52:13.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.721+0000 7fa14bfff640 1 -- 172.21.15.98:0/2343158689 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fa1440522b0 con 0x7fa14c02b4f0 2024-03-29T10:52:13.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.721+0000 7fa14bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:13.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.721+0000 7fa14bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:13.703 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:52:13.703 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:13.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.724+0000 7fa159806640 10 librados: watch_flush enter 2024-03-29T10:52:13.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.725+0000 7fa159806640 10 librados: watch_flush exit 2024-03-29T10:52:13.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.725+0000 7fa159806640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-29T10:52:13.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.725+0000 7fa159806640 10 client.4501.objecter successfully canceled tick 2024-03-29T10:52:13.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.725+0000 7fa159806640 1 -- 172.21.15.98:0/2343158689 >> v1:172.21.15.98:6816/1481799426 conn(0x7fa13406ded0 legacy=0x7fa134070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:13.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.725+0000 7fa159806640 10 monclient: shutdown 2024-03-29T10:52:13.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.725+0000 7fa159806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:13.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.725+0000 7fa159806640 1 -- 172.21.15.98:0/2343158689 >> v1:172.21.15.189:6789/0 conn(0x7fa14c02b4f0 legacy=0x7fa14c02e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:13.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.725+0000 7fa159806640 1 -- 172.21.15.98:0/2343158689 shutdown_connections 2024-03-29T10:52:13.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.725+0000 7fa159806640 1 -- 172.21.15.98:0/2343158689 >> 172.21.15.98:0/2343158689 conn(0x7fa14c005920 msgr2=0x7fa14c038160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:13.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.726+0000 7fa159806640 1 -- 172.21.15.98:0/2343158689 shutdown_connections 2024-03-29T10:52:13.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.726+0000 7fa159806640 1 -- 172.21.15.98:0/2343158689 wait complete. 2024-03-29T10:52:13.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:13.726+0000 7fa159806640 1 librados: shutdown 2024-03-29T10:52:13.723 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:20.530 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:20.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.703+0000 7f73deb3b640 10 monclient: get_monmap_and_config 2024-03-29T10:52:20.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.703+0000 7f73deb3b640 10 monclient: build_initial_monmap 2024-03-29T10:52:20.686 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.704+0000 7f73deb3b640 10 monclient: monmap: 2024-03-29T10:52:20.686 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:20.686 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:20.686 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:52:20.704841+0000 2024-03-29T10:52:20.686 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:52:20.704841+0000 2024-03-29T10:52:20.686 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:20.686 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:20.687 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:20.687 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:52:20.687 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:52:20.687 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:20.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.704+0000 7f73deb3b640 1 Processor -- start 2024-03-29T10:52:20.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.705+0000 7f73deb3b640 1 -- start start 2024-03-29T10:52:20.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.705+0000 7f73deb3b640 10 monclient: init 2024-03-29T10:52:20.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.706+0000 7f73deb3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:20.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.706+0000 7f73deb3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:20.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.706+0000 7f73deb3b640 10 monclient(hunting): picked mon.noname-a con 0x7f73d80b2800 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:20.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.706+0000 7f73deb3b640 10 monclient(hunting): picked mon.noname-c con 0x7f73d80a9c00 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:20.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.706+0000 7f73deb3b640 10 monclient(hunting): picked mon.noname-b con 0x7f73d80b38c0 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:20.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.706+0000 7f73deb3b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d8141eb0 con 0x7f73d80b2800 2024-03-29T10:52:20.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.706+0000 7f73deb3b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d80b22d0 con 0x7f73d80a9c00 2024-03-29T10:52:20.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.706+0000 7f73deb3b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d8142d30 con 0x7f73d80b38c0 2024-03-29T10:52:20.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.706+0000 7f73deb3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:20.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.706+0000 7f73deb3b640 10 monclient(hunting): authenticate will time out at 1360.903425s 2024-03-29T10:52:20.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.706+0000 7f73dd338640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f73d80a9c00 0x7f73d813bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:38230/0 (socket says 172.21.15.98:38230) 2024-03-29T10:52:20.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.706+0000 7f73dd338640 1 -- 172.21.15.98:0/667596325 learned_addr learned my addr 172.21.15.98:0/667596325 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:20.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 1 -- 172.21.15.98:0/667596325 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482969937 0 0) 0x7f73d8142d30 con 0x7f73d80b38c0 2024-03-29T10:52:20.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 10 monclient(hunting): _negotiate 2024-03-29T10:52:20.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 10 monclient(hunting): my global_id is 4535 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 1 -- 172.21.15.98:0/667596325 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73c4003520 con 0x7f73d80b38c0 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 1 -- 172.21.15.98:0/667596325 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361239071 0 0) 0x7f73d8141eb0 con 0x7f73d80b2800 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 10 monclient(hunting): _negotiate 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 10 monclient(hunting): my global_id is 4488 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 1 -- 172.21.15.98:0/667596325 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d8142d30 con 0x7f73d80b2800 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 1 -- 172.21.15.98:0/667596325 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 720901368 0 0) 0x7f73c4003520 con 0x7f73d80b38c0 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 1 -- 172.21.15.98:0/667596325 >> v1:172.21.15.189:6789/0 conn(0x7f73d80a9c00 legacy=0x7f73d813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 1 -- 172.21.15.98:0/667596325 >> v1:172.21.15.98:6789/0 conn(0x7f73d80b2800 legacy=0x7f73d8142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 10 monclient: _finish_hunting 0 2024-03-29T10:52:20.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 1 monclient: found mon.noname-b 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.707+0000 7f73dcb37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 1 -- 172.21.15.98:0/667596325 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d80aaf90 con 0x7f73d80b38c0 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 10 monclient: _finish_auth 0 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 10 monclient: _check_auth_tickets 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73deb3b640 5 monclient: authenticate success, global_id 4535 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73deb3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 1 -- 172.21.15.98:0/667596325 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f73d0001a10 con 0x7f73d80b38c0 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 10 monclient: dump: 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:20.709 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73deb3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 1 -- 172.21.15.98:0/667596325 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73d0001bf0 con 0x7f73d80b38c0 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 1 -- 172.21.15.98:0/667596325 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f73d0001f40 con 0x7f73d80b38c0 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:20.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73dcb37640 10 monclient: dump: 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73deb3b640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.708+0000 7f73deb3b640 4 set_mon_vals no callback set 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.709+0000 7f73deb3b640 10 monclient: shutdown 2024-03-29T10:52:20.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.709+0000 7f73deb3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.709+0000 7f73deb3b640 1 -- 172.21.15.98:0/667596325 >> v1:172.21.15.98:6790/0 conn(0x7f73d80b38c0 legacy=0x7f73d814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.709+0000 7f73deb3b640 1 -- 172.21.15.98:0/667596325 shutdown_connections 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.709+0000 7f73deb3b640 1 -- 172.21.15.98:0/667596325 >> 172.21.15.98:0/667596325 conn(0x7f73d8066a00 msgr2=0x7f73d8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.709+0000 7f73deb3b640 1 -- 172.21.15.98:0/667596325 shutdown_connections 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.709+0000 7f73deb3b640 1 -- 172.21.15.98:0/667596325 wait complete. 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.710+0000 7f73deb3b640 10 monclient: build_initial_monmap 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.710+0000 7f73deb3b640 10 monclient: monmap: 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:20.712 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.710+0000 7f73deb3b640 1 librados: starting msgr at 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.710+0000 7f73deb3b640 1 librados: starting objecter 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 1 Processor -- start 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 1 -- start start 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 1 librados: setting wanted keys 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 1 librados: calling monclient init 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 10 monclient: init 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 10 monclient(hunting): picked mon.noname-a con 0x7f73d80a9c00 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 10 monclient(hunting): picked mon.noname-b con 0x7f73d80b2800 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 10 monclient(hunting): picked mon.noname-c con 0x7f73d80b38c0 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:20.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d8071270 con 0x7f73d80a9c00 2024-03-29T10:52:20.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d806fd50 con 0x7f73d80b2800 2024-03-29T10:52:20.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d8070030 con 0x7f73d80b38c0 2024-03-29T10:52:20.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:20.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.711+0000 7f73deb3b640 10 monclient(hunting): authenticate will time out at 1360.908803s 2024-03-29T10:52:20.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73de33a640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f73d80b38c0 0x7f73d81e63f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:34756/0 (socket says 172.21.15.98:34756) 2024-03-29T10:52:20.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73de33a640 1 -- 172.21.15.98:0/1485611785 learned_addr learned my addr 172.21.15.98:0/1485611785 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:20.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 10 client.?.objecter ms_handle_connect 0x7f73d80b38c0 2024-03-29T10:52:20.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:20.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 10 client.?.objecter ms_handle_connect 0x7f73d80a9c00 2024-03-29T10:52:20.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:20.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697088604 0 0) 0x7f73d8070030 con 0x7f73d80b38c0 2024-03-29T10:52:20.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:20.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 10 monclient(hunting): my global_id is 4538 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73bc004eb0 con 0x7f73d80b38c0 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150999238 0 0) 0x7f73d8071270 con 0x7f73d80a9c00 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 10 monclient(hunting): my global_id is 4491 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.712+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d8070030 con 0x7f73d80a9c00 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1097029963 0 0) 0x7f73d8070030 con 0x7f73d80a9c00 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73bc0064f0 con 0x7f73d80a9c00 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1839705547 0 0) 0x7f73bc004eb0 con 0x7f73d80b38c0 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73bc006a80 con 0x7f73d80b38c0 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f73d4001930 con 0x7f73d80a9c00 2024-03-29T10:52:20.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:20.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:20.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 10 monclient(hunting): dump: 2024-03-29T10:52:20.717 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:20.717 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:20.717 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:20.717 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:20.717 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:20.717 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f73d0001f40 con 0x7f73d80b38c0 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 10 monclient(hunting): dump: 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:20.718 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 927359321 0 0) 0x7f73bc006a80 con 0x7f73d80b38c0 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 >> v1:172.21.15.189:6789/0 conn(0x7f73d80b2800 legacy=0x7f73d813d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 >> v1:172.21.15.98:6789/0 conn(0x7f73d80a9c00 legacy=0x7f73d806de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 1 monclient: found mon.c 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d81388f0 con 0x7f73d80b38c0 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 10 monclient: _finish_auth 0 2024-03-29T10:52:20.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:52:20.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:20.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73deb3b640 5 monclient: authenticate success, global_id 4538 2024-03-29T10:52:20.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73deb3b640 10 monclient: _renew_subs 2024-03-29T10:52:20.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73deb3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:20.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73deb3b640 1 -- 172.21.15.98:0/1485611785 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d806ee10 con 0x7f73d80b38c0 2024-03-29T10:52:20.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73deb3b640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:20.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73deb3b640 10 monclient: _renew_subs 2024-03-29T10:52:20.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73deb3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:20.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.713+0000 7f73deb3b640 1 -- 172.21.15.98:0/1485611785 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d80704c0 con 0x7f73d80b38c0 2024-03-29T10:52:20.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.714+0000 7f73deb3b640 1 librados: init done 2024-03-29T10:52:20.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.714+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73d0003c20 con 0x7f73d80b38c0 2024-03-29T10:52:20.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.714+0000 7f73ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:20.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.714+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f73d0003fb0 con 0x7f73d80b38c0 2024-03-29T10:52:20.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.714+0000 7f73ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:20.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.714+0000 7f73ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:20.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.714+0000 7f73ce7fc640 10 monclient: dump: 2024-03-29T10:52:20.721 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:20.721 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:20.721 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:20.721 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:20.721 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:20.721 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:20.721 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.714+0000 7f73cffff640 4 set_mon_vals no callback set 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.716+0000 7f73deb3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.716+0000 7f73deb3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.716+0000 7f73deb3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.716+0000 7f73deb3b640 1 -- 172.21.15.98:0/1485611785 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c0005080 con 0x7f73d80b38c0 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.716+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f73d0004110 con 0x7f73d80b38c0 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.717+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f73d0084e80 con 0x7f73d80b38c0 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.717+0000 7f73ce7fc640 10 client.4538.objecter ms_dispatch 0x7f73d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.717+0000 7f73ce7fc640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.717+0000 7f73ce7fc640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:20.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.717+0000 7f73ce7fc640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-29T10:52:20.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.717+0000 7f73ddb39640 10 monclient: get_auth_request con 0x7f73bc06e1a0 auth_method 0 2024-03-29T10:52:20.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.717+0000 7f73ce7fc640 10 client.4538.objecter ms_handle_connect 0x7f73bc06e1a0 2024-03-29T10:52:20.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.725+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73d0050760 con 0x7f73d80b38c0 2024-03-29T10:52:20.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.725+0000 7f73ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:20.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:20.725+0000 7f73ce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:21.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.048+0000 7f73deb3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:21.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.048+0000 7f73deb3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:21.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.048+0000 7f73deb3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:21.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.048+0000 7f73deb3b640 1 -- 172.21.15.98:0/1485611785 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73c0005330 con 0x7f73d80b38c0 2024-03-29T10:52:21.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.050+0000 7f73ce7fc640 1 -- 172.21.15.98:0/1485611785 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f73d0052690 con 0x7f73d80b38c0 2024-03-29T10:52:21.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.050+0000 7f73ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:21.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.050+0000 7f73ce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:21.033 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:52:21.033 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:21.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.054+0000 7f73deb3b640 10 librados: watch_flush enter 2024-03-29T10:52:21.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.054+0000 7f73deb3b640 10 librados: watch_flush exit 2024-03-29T10:52:21.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.054+0000 7f73deb3b640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-29T10:52:21.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.055+0000 7f73deb3b640 10 client.4538.objecter successfully canceled tick 2024-03-29T10:52:21.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.055+0000 7f73deb3b640 1 -- 172.21.15.98:0/1485611785 >> v1:172.21.15.98:6816/1481799426 conn(0x7f73bc06e1a0 legacy=0x7f73bc070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:21.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.055+0000 7f73deb3b640 10 monclient: shutdown 2024-03-29T10:52:21.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.055+0000 7f73deb3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:21.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.055+0000 7f73deb3b640 1 -- 172.21.15.98:0/1485611785 >> v1:172.21.15.98:6790/0 conn(0x7f73d80b38c0 legacy=0x7f73d81e63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:21.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.055+0000 7f73deb3b640 1 -- 172.21.15.98:0/1485611785 shutdown_connections 2024-03-29T10:52:21.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.055+0000 7f73deb3b640 1 -- 172.21.15.98:0/1485611785 >> 172.21.15.98:0/1485611785 conn(0x7f73d8066a00 msgr2=0x7f73d80b7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:21.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.057+0000 7f73deb3b640 1 -- 172.21.15.98:0/1485611785 shutdown_connections 2024-03-29T10:52:21.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.057+0000 7f73deb3b640 1 -- 172.21.15.98:0/1485611785 wait complete. 2024-03-29T10:52:21.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:21.057+0000 7f73deb3b640 1 librados: shutdown 2024-03-29T10:52:21.059 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:27.867 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:28.019 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.038+0000 7f659dfaf640 10 monclient: get_monmap_and_config 2024-03-29T10:52:28.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.038+0000 7f659dfaf640 10 monclient: build_initial_monmap 2024-03-29T10:52:28.020 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.038+0000 7f659dfaf640 10 monclient: monmap: 2024-03-29T10:52:28.020 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:28.020 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:28.020 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:52:28.038975+0000 2024-03-29T10:52:28.020 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:52:28.038975+0000 2024-03-29T10:52:28.020 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:28.020 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:28.020 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:28.020 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:52:28.020 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:52:28.020 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:28.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.039+0000 7f659dfaf640 1 Processor -- start 2024-03-29T10:52:28.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.039+0000 7f659dfaf640 1 -- start start 2024-03-29T10:52:28.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.039+0000 7f659dfaf640 10 monclient: init 2024-03-29T10:52:28.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.040+0000 7f659dfaf640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:28.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.040+0000 7f659dfaf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:28.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.040+0000 7f659dfaf640 10 monclient(hunting): picked mon.noname-c con 0x7f65981457f0 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:28.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.040+0000 7f659dfaf640 10 monclient(hunting): picked mon.noname-a con 0x7f659813f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:28.022 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.040+0000 7f659dfaf640 10 monclient(hunting): picked mon.noname-b con 0x7f65980a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:28.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.040+0000 7f659dfaf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65980aaef0 con 0x7f659813f650 2024-03-29T10:52:28.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.040+0000 7f659dfaf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6598145bd0 con 0x7f65981457f0 2024-03-29T10:52:28.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.040+0000 7f659dfaf640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6598145eb0 con 0x7f65980a9d70 2024-03-29T10:52:28.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.040+0000 7f659dfaf640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:28.023 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.040+0000 7f659dfaf640 10 monclient(hunting): authenticate will time out at 1368.237351s 2024-03-29T10:52:28.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.043+0000 7f6597fff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f65980a9d70 0x7f659814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:44158/0 (socket says 172.21.15.98:44158) 2024-03-29T10:52:28.024 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.043+0000 7f6597fff640 1 -- 172.21.15.98:0/2281315800 learned_addr learned my addr 172.21.15.98:0/2281315800 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:28.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.043+0000 7f65967fc640 1 -- 172.21.15.98:0/2281315800 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404939022 0 0) 0x7f6598145eb0 con 0x7f65980a9d70 2024-03-29T10:52:28.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.043+0000 7f65967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:52:28.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.043+0000 7f65967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:28.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.043+0000 7f65967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:28.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.043+0000 7f65967fc640 10 monclient(hunting): my global_id is 4541 2024-03-29T10:52:28.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.043+0000 7f65967fc640 1 -- 172.21.15.98:0/2281315800 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6584003520 con 0x7f65980a9d70 2024-03-29T10:52:28.025 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 1 -- 172.21.15.98:0/2281315800 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 253902552 0 0) 0x7f6584003520 con 0x7f65980a9d70 2024-03-29T10:52:28.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 1 -- 172.21.15.98:0/2281315800 >> v1:172.21.15.189:6789/0 conn(0x7f65981457f0 legacy=0x7f659813ef40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:52:28.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 1 -- 172.21.15.98:0/2281315800 >> v1:172.21.15.98:6789/0 conn(0x7f659813f650 legacy=0x7f65980b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:28.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 10 monclient: _finish_hunting 0 2024-03-29T10:52:28.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 1 monclient: found mon.noname-b 2024-03-29T10:52:28.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:28.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:52:28.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 1 -- 172.21.15.98:0/2281315800 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6598146340 con 0x7f65980a9d70 2024-03-29T10:52:28.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 10 monclient: _finish_auth 0 2024-03-29T10:52:28.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 10 monclient: _check_auth_tickets 2024-03-29T10:52:28.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:28.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 1 -- 172.21.15.98:0/2281315800 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f6580001690 con 0x7f65980a9d70 2024-03-29T10:52:28.026 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:28.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:28.027 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.044+0000 7f65967fc640 10 monclient: dump: 2024-03-29T10:52:28.027 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:28.027 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:28.027 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:28.027 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:28.027 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:28.027 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:28.027 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:28.027 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:28.027 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:28.027 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:28.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.046+0000 7f659dfaf640 5 monclient: authenticate success, global_id 4541 2024-03-29T10:52:28.028 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.046+0000 7f659dfaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:28.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.046+0000 7f65967fc640 1 -- 172.21.15.98:0/2281315800 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6580001860 con 0x7f65980a9d70 2024-03-29T10:52:28.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.047+0000 7f65967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:28.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.047+0000 7f659dfaf640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:28.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.047+0000 7f659dfaf640 4 set_mon_vals no callback set 2024-03-29T10:52:28.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.047+0000 7f65967fc640 1 -- 172.21.15.98:0/2281315800 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f6580001cb0 con 0x7f65980a9d70 2024-03-29T10:52:28.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.047+0000 7f65967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:28.030 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.047+0000 7f65967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.047+0000 7f65967fc640 10 monclient: dump: 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.047+0000 7f659dfaf640 10 monclient: shutdown 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.047+0000 7f659dfaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:28.031 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.047+0000 7f659dfaf640 1 -- 172.21.15.98:0/2281315800 >> v1:172.21.15.98:6790/0 conn(0x7f65980a9d70 legacy=0x7f659814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:28.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.048+0000 7f659dfaf640 1 -- 172.21.15.98:0/2281315800 shutdown_connections 2024-03-29T10:52:28.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.048+0000 7f659dfaf640 1 -- 172.21.15.98:0/2281315800 >> 172.21.15.98:0/2281315800 conn(0x7f6598066a00 msgr2=0x7f6598066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:28.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.049+0000 7f659dfaf640 1 -- 172.21.15.98:0/2281315800 shutdown_connections 2024-03-29T10:52:28.033 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.049+0000 7f659dfaf640 1 -- 172.21.15.98:0/2281315800 wait complete. 2024-03-29T10:52:28.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.050+0000 7f659dfaf640 10 monclient: build_initial_monmap 2024-03-29T10:52:28.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.050+0000 7f659dfaf640 10 monclient: monmap: 2024-03-29T10:52:28.034 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:28.034 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:28.034 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:52:28.034 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:52:28.034 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:28.034 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:28.034 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:28.034 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:52:28.034 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:52:28.034 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:28.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.050+0000 7f659dfaf640 1 librados: starting msgr at 2024-03-29T10:52:28.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.050+0000 7f659dfaf640 1 librados: starting objecter 2024-03-29T10:52:28.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.050+0000 7f659dfaf640 1 Processor -- start 2024-03-29T10:52:28.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.051+0000 7f659dfaf640 1 -- start start 2024-03-29T10:52:28.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.051+0000 7f659dfaf640 1 librados: setting wanted keys 2024-03-29T10:52:28.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.051+0000 7f659dfaf640 1 librados: calling monclient init 2024-03-29T10:52:28.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.051+0000 7f659dfaf640 10 monclient: init 2024-03-29T10:52:28.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.051+0000 7f659dfaf640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:28.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.051+0000 7f659dfaf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:28.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.051+0000 7f659dfaf640 10 monclient(hunting): picked mon.noname-a con 0x7f65980a9d70 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:28.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.052+0000 7f659dfaf640 10 monclient(hunting): picked mon.noname-c con 0x7f659813f650 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:28.036 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.052+0000 7f6596ffd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f659813f650 0x7f6598140580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:44754/0 (socket says 172.21.15.98:44754) 2024-03-29T10:52:28.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.055+0000 7f6596ffd640 1 -- 172.21.15.98:0/1714513866 learned_addr learned my addr 172.21.15.98:0/1714513866 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:28.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 client.?.objecter ms_handle_connect 0x7f659813f650 2024-03-29T10:52:28.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f659dfaf640 10 monclient(hunting): picked mon.noname-b con 0x7f65981457f0 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:28.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:28.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 client.?.objecter ms_handle_connect 0x7f65981457f0 2024-03-29T10:52:28.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:28.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f659dfaf640 1 -- 172.21.15.98:0/1714513866 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659813aed0 con 0x7f65980a9d70 2024-03-29T10:52:28.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f659dfaf640 1 -- 172.21.15.98:0/1714513866 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659813b120 con 0x7f65981457f0 2024-03-29T10:52:28.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f659dfaf640 1 -- 172.21.15.98:0/1714513866 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659813bc00 con 0x7f659813f650 2024-03-29T10:52:28.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f659dfaf640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:28.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f659dfaf640 10 monclient(hunting): authenticate will time out at 1368.253171s 2024-03-29T10:52:28.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283118069 0 0) 0x7f659813b120 con 0x7f65981457f0 2024-03-29T10:52:28.041 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:28.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:28.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:28.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 monclient(hunting): my global_id is 4507 2024-03-29T10:52:28.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6578004eb0 con 0x7f65981457f0 2024-03-29T10:52:28.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249920365 0 0) 0x7f659813bc00 con 0x7f659813f650 2024-03-29T10:52:28.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:28.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:28.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:28.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 monclient(hunting): my global_id is 4544 2024-03-29T10:52:28.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f659813b120 con 0x7f659813f650 2024-03-29T10:52:28.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3968181912 0 0) 0x7f6578004eb0 con 0x7f65981457f0 2024-03-29T10:52:28.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f659813bc00 con 0x7f65981457f0 2024-03-29T10:52:28.042 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f65800019d0 con 0x7f65981457f0 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 monclient(hunting): dump: 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:28.043 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:28.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.056+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4042896382 0 0) 0x7f659813b120 con 0x7f659813f650 2024-03-29T10:52:28.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6578004eb0 con 0x7f659813f650 2024-03-29T10:52:28.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f65880019f0 con 0x7f659813f650 2024-03-29T10:52:28.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:28.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:28.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 10 monclient(hunting): dump: 2024-03-29T10:52:28.044 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:28.044 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:28.044 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:28.044 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:28.044 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:28.044 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1706668856 0 0) 0x7f659813bc00 con 0x7f65981457f0 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 >> v1:172.21.15.98:6790/0 conn(0x7f659813f650 legacy=0x7f6598140580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 >> v1:172.21.15.98:6789/0 conn(0x7f65980a9d70 legacy=0x7f6598138f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 1 monclient: found mon.b 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659813b820 con 0x7f65981457f0 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 10 monclient: _finish_auth 0 2024-03-29T10:52:28.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:28.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.057+0000 7f658ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:28.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f659dfaf640 5 monclient: authenticate success, global_id 4507 2024-03-29T10:52:28.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f659dfaf640 10 monclient: _renew_subs 2024-03-29T10:52:28.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f659dfaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:52:28.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f659dfaf640 1 -- 172.21.15.98:0/1714513866 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6598139b00 con 0x7f65981457f0 2024-03-29T10:52:28.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f659dfaf640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:28.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f659dfaf640 10 monclient: _renew_subs 2024-03-29T10:52:28.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f659dfaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:52:28.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f659dfaf640 1 -- 172.21.15.98:0/1714513866 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65980b1840 con 0x7f65981457f0 2024-03-29T10:52:28.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f659dfaf640 1 librados: init done 2024-03-29T10:52:28.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6580001c70 con 0x7f65981457f0 2024-03-29T10:52:28.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f658ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:28.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f6580004d90 con 0x7f65981457f0 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f658ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f658ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f658ffff640 10 monclient: dump: 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:28.047 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.058+0000 7f6595ffb640 4 set_mon_vals no callback set 2024-03-29T10:52:28.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.059+0000 7f659dfaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:28.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.059+0000 7f659dfaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:28.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.059+0000 7f659dfaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:52:28.048 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.059+0000 7f659dfaf640 1 -- 172.21.15.98:0/1714513866 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657c005080 con 0x7f65981457f0 2024-03-29T10:52:28.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.061+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f658001ad60 con 0x7f65981457f0 2024-03-29T10:52:28.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.062+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f6580054820 con 0x7f65981457f0 2024-03-29T10:52:28.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.062+0000 7f658ffff640 10 client.4507.objecter ms_dispatch 0x7f6598000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:28.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.062+0000 7f658ffff640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:28.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.062+0000 7f658ffff640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:28.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.062+0000 7f658ffff640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-29T10:52:28.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.063+0000 7f65977fe640 10 monclient: get_auth_request con 0x7f657806de40 auth_method 0 2024-03-29T10:52:28.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.064+0000 7f658ffff640 10 client.4507.objecter ms_handle_connect 0x7f657806de40 2024-03-29T10:52:28.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.072+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6580050ac0 con 0x7f65981457f0 2024-03-29T10:52:28.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.072+0000 7f658ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:28.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.072+0000 7f658ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:28.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.398+0000 7f659dfaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:28.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.398+0000 7f659dfaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:28.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.399+0000 7f659dfaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:52:28.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.399+0000 7f659dfaf640 1 -- 172.21.15.98:0/1714513866 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f657c005330 con 0x7f65981457f0 2024-03-29T10:52:28.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.399+0000 7f658ffff640 1 -- 172.21.15.98:0/1714513866 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f6580050c60 con 0x7f65981457f0 2024-03-29T10:52:28.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.399+0000 7f658ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:28.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.399+0000 7f658ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:28.382 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:52:28.383 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:28.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.404+0000 7f659dfaf640 10 librados: watch_flush enter 2024-03-29T10:52:28.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.404+0000 7f659dfaf640 10 librados: watch_flush exit 2024-03-29T10:52:28.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.404+0000 7f659dfaf640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-29T10:52:28.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.404+0000 7f659dfaf640 10 client.4507.objecter successfully canceled tick 2024-03-29T10:52:28.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.406+0000 7f659dfaf640 1 -- 172.21.15.98:0/1714513866 >> v1:172.21.15.98:6816/1481799426 conn(0x7f657806de40 legacy=0x7f6578070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:28.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.406+0000 7f659dfaf640 10 monclient: shutdown 2024-03-29T10:52:28.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.406+0000 7f659dfaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:28.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.406+0000 7f659dfaf640 1 -- 172.21.15.98:0/1714513866 >> v1:172.21.15.189:6789/0 conn(0x7f65981457f0 legacy=0x7f6598140c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:28.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.406+0000 7f659dfaf640 1 -- 172.21.15.98:0/1714513866 shutdown_connections 2024-03-29T10:52:28.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.406+0000 7f659dfaf640 1 -- 172.21.15.98:0/1714513866 >> 172.21.15.98:0/1714513866 conn(0x7f6598066a00 msgr2=0x7f65980b5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:28.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.406+0000 7f659dfaf640 1 -- 172.21.15.98:0/1714513866 shutdown_connections 2024-03-29T10:52:28.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.407+0000 7f659dfaf640 1 -- 172.21.15.98:0/1714513866 wait complete. 2024-03-29T10:52:28.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:28.408+0000 7f659dfaf640 1 librados: shutdown 2024-03-29T10:52:28.408 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:35.215 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:35.363 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.381+0000 7f3a1b54d640 10 monclient: get_monmap_and_config 2024-03-29T10:52:35.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.381+0000 7f3a1b54d640 10 monclient: build_initial_monmap 2024-03-29T10:52:35.364 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.381+0000 7f3a1b54d640 10 monclient: monmap: 2024-03-29T10:52:35.364 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:35.364 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:35.364 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:52:35.382658+0000 2024-03-29T10:52:35.364 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:52:35.382658+0000 2024-03-29T10:52:35.364 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:35.364 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:35.364 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:35.364 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:52:35.364 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:52:35.364 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:35.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.383+0000 7f3a1b54d640 1 Processor -- start 2024-03-29T10:52:35.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.384+0000 7f3a1b54d640 1 -- start start 2024-03-29T10:52:35.365 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.384+0000 7f3a1b54d640 10 monclient: init 2024-03-29T10:52:35.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.385+0000 7f3a1b54d640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:35.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.385+0000 7f3a1b54d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:35.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.385+0000 7f3a1b54d640 10 monclient(hunting): picked mon.noname-b con 0x7f3a140b4100 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:35.367 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.385+0000 7f3a1b54d640 10 monclient(hunting): picked mon.noname-c con 0x7f3a1413ee90 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:35.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.385+0000 7f3a1b54d640 10 monclient(hunting): picked mon.noname-a con 0x7f3a140a9ac0 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:35.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.385+0000 7f3a1b54d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a140af6f0 con 0x7f3a140a9ac0 2024-03-29T10:52:35.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.385+0000 7f3a1b54d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a140aac80 con 0x7f3a1413ee90 2024-03-29T10:52:35.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.385+0000 7f3a1b54d640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a140aaf60 con 0x7f3a140b4100 2024-03-29T10:52:35.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.385+0000 7f3a1b54d640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:35.368 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.385+0000 7f3a1b54d640 10 monclient(hunting): authenticate will time out at 1375.582267s 2024-03-29T10:52:35.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.387+0000 7f3a192e9640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f3a140b4100 0x7f3a140adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:41528/0 (socket says 172.21.15.98:41528) 2024-03-29T10:52:35.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.387+0000 7f3a192e9640 1 -- 172.21.15.98:0/2913692802 learned_addr learned my addr 172.21.15.98:0/2913692802 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:35.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 1 -- 172.21.15.98:0/2913692802 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817957400 0 0) 0x7f3a140af6f0 con 0x7f3a140a9ac0 2024-03-29T10:52:35.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:52:35.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:35.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:35.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient(hunting): my global_id is 4497 2024-03-29T10:52:35.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 1 -- 172.21.15.98:0/2913692802 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a00003520 con 0x7f3a140a9ac0 2024-03-29T10:52:35.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 1 -- 172.21.15.98:0/2913692802 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2570244030 0 0) 0x7f3a00003520 con 0x7f3a140a9ac0 2024-03-29T10:52:35.374 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 1 -- 172.21.15.98:0/2913692802 >> v1:172.21.15.98:6790/0 conn(0x7f3a140b4100 legacy=0x7f3a140adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 1 -- 172.21.15.98:0/2913692802 >> v1:172.21.15.189:6789/0 conn(0x7f3a1413ee90 legacy=0x7f3a1413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 1 monclient: found mon.noname-a 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 1 -- 172.21.15.98:0/2913692802 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1413e970 con 0x7f3a140a9ac0 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient: _finish_auth 0 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a1b54d640 5 monclient: authenticate success, global_id 4497 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a1b54d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 1 -- 172.21.15.98:0/2913692802 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3a0c001a10 con 0x7f3a140a9ac0 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:52:35.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient: dump: 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a1b54d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 1 -- 172.21.15.98:0/2913692802 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a0c001ca0 con 0x7f3a140a9ac0 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a1b54d640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a1b54d640 4 set_mon_vals no callback set 2024-03-29T10:52:35.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 1 -- 172.21.15.98:0/2913692802 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3a0c002030 con 0x7f3a140a9ac0 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.390+0000 7f3a0bfff640 10 monclient: dump: 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.391+0000 7f3a1b54d640 10 monclient: shutdown 2024-03-29T10:52:35.377 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.391+0000 7f3a1b54d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:35.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.391+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2913692802 >> v1:172.21.15.98:6789/0 conn(0x7f3a140a9ac0 legacy=0x7f3a14142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:35.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.391+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2913692802 shutdown_connections 2024-03-29T10:52:35.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.391+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2913692802 >> 172.21.15.98:0/2913692802 conn(0x7f3a140b35e0 msgr2=0x7f3a140b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:35.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.392+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2913692802 shutdown_connections 2024-03-29T10:52:35.378 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.392+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2913692802 wait complete. 2024-03-29T10:52:35.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.393+0000 7f3a1b54d640 10 monclient: build_initial_monmap 2024-03-29T10:52:35.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.393+0000 7f3a1b54d640 10 monclient: monmap: 2024-03-29T10:52:35.384 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:35.385 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:35.385 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:52:35.385 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:52:35.385 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:35.385 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:35.385 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:35.385 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:52:35.385 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:52:35.385 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:35.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.393+0000 7f3a1b54d640 1 librados: starting msgr at 2024-03-29T10:52:35.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.393+0000 7f3a1b54d640 1 librados: starting objecter 2024-03-29T10:52:35.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.393+0000 7f3a1b54d640 1 Processor -- start 2024-03-29T10:52:35.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.393+0000 7f3a1b54d640 1 -- start start 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.393+0000 7f3a1b54d640 1 librados: setting wanted keys 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.393+0000 7f3a1b54d640 1 librados: calling monclient init 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.393+0000 7f3a1b54d640 10 monclient: init 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.394+0000 7f3a1b54d640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.394+0000 7f3a1b54d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.394+0000 7f3a1b54d640 10 monclient(hunting): picked mon.noname-b con 0x7f3a140a9ac0 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.394+0000 7f3a1b54d640 10 monclient(hunting): picked mon.noname-a con 0x7f3a140b4100 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.394+0000 7f3a1b54d640 10 monclient(hunting): picked mon.noname-c con 0x7f3a1413ee90 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.394+0000 7f3a1b54d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a14142100 con 0x7f3a140b4100 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.394+0000 7f3a1b54d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a14140600 con 0x7f3a140a9ac0 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.394+0000 7f3a1b54d640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a141416b0 con 0x7f3a1413ee90 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.394+0000 7f3a1b54d640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.394+0000 7f3a1b54d640 10 monclient(hunting): authenticate will time out at 1375.591903s 2024-03-29T10:52:35.386 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.396+0000 7f3a19aea640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f3a1413ee90 0x7f3a14139e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:42946/0 (socket says 172.21.15.98:42946) 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.396+0000 7f3a19aea640 1 -- 172.21.15.98:0/2407115745 learned_addr learned my addr 172.21.15.98:0/2407115745 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.396+0000 7f3a09ffb640 10 client.?.objecter ms_handle_connect 0x7f3a1413ee90 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.396+0000 7f3a09ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.396+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345427500 0 0) 0x7f3a141416b0 con 0x7f3a1413ee90 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.396+0000 7f3a09ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.396+0000 7f3a09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.396+0000 7f3a09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.396+0000 7f3a09ffb640 10 monclient(hunting): my global_id is 4547 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.396+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f8004eb0 con 0x7f3a1413ee90 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.397+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1023169531 0 0) 0x7f39f8004eb0 con 0x7f3a1413ee90 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.397+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39f8005300 con 0x7f3a1413ee90 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.397+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3a0c001a90 con 0x7f3a1413ee90 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.397+0000 7f3a09ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.397+0000 7f3a09ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:35.387 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.397+0000 7f3a09ffb640 10 monclient(hunting): dump: 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.397+0000 7f3a09ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.398+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3672340494 0 0) 0x7f39f8005300 con 0x7f3a1413ee90 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.398+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 >> v1:172.21.15.189:6789/0 conn(0x7f3a140a9ac0 legacy=0x7f3a14140fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.398+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 >> v1:172.21.15.98:6789/0 conn(0x7f3a140b4100 legacy=0x7f3a14139740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:52:35.388 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.398+0000 7f3a09ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.398+0000 7f3a09ffb640 1 monclient: found mon.c 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.398+0000 7f3a09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a09ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a14142330 con 0x7f3a1413ee90 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a09ffb640 10 monclient: _finish_auth 0 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a09ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a1b54d640 5 monclient: authenticate success, global_id 4547 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a1b54d640 10 monclient: _renew_subs 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a1b54d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2407115745 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a1406e8d0 con 0x7f3a1413ee90 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a1b54d640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a1b54d640 10 monclient: _renew_subs 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a1b54d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:35.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2407115745 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a141e7810 con 0x7f3a1413ee90 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a1b54d640 1 librados: init done 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a0c003d60 con 0x7f3a1413ee90 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a09ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.399+0000 7f3a0affd640 4 set_mon_vals no callback set 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.400+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3a0c001a90 con 0x7f3a1413ee90 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.400+0000 7f3a09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.400+0000 7f3a09ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.400+0000 7f3a09ffb640 10 monclient: dump: 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:35.390 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.401+0000 7f3a1b54d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.401+0000 7f3a1b54d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.401+0000 7f3a1b54d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.401+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2407115745 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f4005080 con 0x7f3a1413ee90 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.402+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f3a0c004050 con 0x7f3a1413ee90 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.403+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f3a0c084a40 con 0x7f3a1413ee90 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.403+0000 7f3a09ffb640 10 client.4547.objecter ms_dispatch 0x7f3a14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.403+0000 7f3a09ffb640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.403+0000 7f3a09ffb640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.403+0000 7f3a09ffb640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T10:52:35.391 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.403+0000 7f3a192e9640 10 monclient: get_auth_request con 0x7f39f806e260 auth_method 0 2024-03-29T10:52:35.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.404+0000 7f3a09ffb640 10 client.4547.objecter ms_handle_connect 0x7f39f806e260 2024-03-29T10:52:35.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.412+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a0c050380 con 0x7f3a1413ee90 2024-03-29T10:52:35.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.412+0000 7f3a09ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:35.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.412+0000 7f3a09ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:35.722 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.741+0000 7f3a1b54d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:35.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.741+0000 7f3a1b54d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:35.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.741+0000 7f3a1b54d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:35.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.741+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2407115745 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39f4005330 con 0x7f3a1413ee90 2024-03-29T10:52:35.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.742+0000 7f3a09ffb640 1 -- 172.21.15.98:0/2407115745 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f3a0c0840f0 con 0x7f3a1413ee90 2024-03-29T10:52:35.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.742+0000 7f3a09ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:35.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.742+0000 7f3a09ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:35.724 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:52:35.724 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:35.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.746+0000 7f3a1b54d640 10 librados: watch_flush enter 2024-03-29T10:52:35.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.746+0000 7f3a1b54d640 10 librados: watch_flush exit 2024-03-29T10:52:35.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.746+0000 7f3a1b54d640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T10:52:35.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.746+0000 7f3a1b54d640 10 client.4547.objecter successfully canceled tick 2024-03-29T10:52:35.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.746+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2407115745 >> v1:172.21.15.98:6816/1481799426 conn(0x7f39f806e260 legacy=0x7f39f8070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:35.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.746+0000 7f3a1b54d640 10 monclient: shutdown 2024-03-29T10:52:35.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.746+0000 7f3a1b54d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:35.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.746+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2407115745 >> v1:172.21.15.98:6790/0 conn(0x7f3a1413ee90 legacy=0x7f3a14139e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:35.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.746+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2407115745 shutdown_connections 2024-03-29T10:52:35.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.746+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2407115745 >> 172.21.15.98:0/2407115745 conn(0x7f3a140b35e0 msgr2=0x7f3a140adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:35.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.746+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2407115745 shutdown_connections 2024-03-29T10:52:35.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.746+0000 7f3a1b54d640 1 -- 172.21.15.98:0/2407115745 wait complete. 2024-03-29T10:52:35.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:35.746+0000 7f3a1b54d640 1 librados: shutdown 2024-03-29T10:52:35.744 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:42.551 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:42.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.717+0000 7f4b29220640 10 monclient: get_monmap_and_config 2024-03-29T10:52:42.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.717+0000 7f4b29220640 10 monclient: build_initial_monmap 2024-03-29T10:52:42.701 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.717+0000 7f4b29220640 10 monclient: monmap: 2024-03-29T10:52:42.701 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:42.701 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:42.701 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:52:42.719737+0000 2024-03-29T10:52:42.701 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:52:42.719737+0000 2024-03-29T10:52:42.702 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:42.702 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:42.702 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:42.702 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:52:42.702 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:52:42.702 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:42.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.718+0000 7f4b29220640 1 Processor -- start 2024-03-29T10:52:42.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.718+0000 7f4b29220640 1 -- start start 2024-03-29T10:52:42.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.718+0000 7f4b29220640 10 monclient: init 2024-03-29T10:52:42.703 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.719+0000 7f4b29220640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:42.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.719+0000 7f4b29220640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:52:42.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.719+0000 7f4b29220640 10 monclient(hunting): picked mon.noname-a con 0x7f4b240b4100 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:42.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.719+0000 7f4b29220640 10 monclient(hunting): picked mon.noname-b con 0x7f4b2413ee90 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:42.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.719+0000 7f4b29220640 10 monclient(hunting): picked mon.noname-c con 0x7f4b240a9ac0 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:42.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.719+0000 7f4b29220640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b240af6f0 con 0x7f4b240b4100 2024-03-29T10:52:42.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.719+0000 7f4b29220640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b240aac80 con 0x7f4b240a9ac0 2024-03-29T10:52:42.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.719+0000 7f4b29220640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b240aaf60 con 0x7f4b2413ee90 2024-03-29T10:52:42.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.719+0000 7f4b29220640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:42.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.719+0000 7f4b29220640 10 monclient(hunting): authenticate will time out at 1382.917751s 2024-03-29T10:52:42.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.721+0000 7f4b2259c640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f4b2413ee90 0x7f4b24146ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:52878/0 (socket says 172.21.15.98:52878) 2024-03-29T10:52:42.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.721+0000 7f4b2259c640 1 -- 172.21.15.98:0/1014315590 learned_addr learned my addr 172.21.15.98:0/1014315590 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:42.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.722+0000 7f4b21d9b640 1 -- 172.21.15.98:0/1014315590 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753440786 0 0) 0x7f4b240aaf60 con 0x7f4b2413ee90 2024-03-29T10:52:42.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.722+0000 7f4b21d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:52:42.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.722+0000 7f4b21d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:42.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.722+0000 7f4b21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:42.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.722+0000 7f4b21d9b640 10 monclient(hunting): my global_id is 4550 2024-03-29T10:52:42.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.722+0000 7f4b21d9b640 1 -- 172.21.15.98:0/1014315590 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b0c003520 con 0x7f4b2413ee90 2024-03-29T10:52:42.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.722+0000 7f4b21d9b640 1 -- 172.21.15.98:0/1014315590 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1773678570 0 0) 0x7f4b0c003520 con 0x7f4b2413ee90 2024-03-29T10:52:42.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.722+0000 7f4b21d9b640 1 -- 172.21.15.98:0/1014315590 >> v1:172.21.15.189:6789/0 conn(0x7f4b240a9ac0 legacy=0x7f4b2414a210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:42.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.723+0000 7f4b21d9b640 1 -- 172.21.15.98:0/1014315590 >> v1:172.21.15.98:6789/0 conn(0x7f4b240b4100 legacy=0x7f4b2413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:42.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b21d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:52:42.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b21d9b640 1 monclient: found mon.noname-b 2024-03-29T10:52:42.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b21d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:42.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b21d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:52:42.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b21d9b640 1 -- 172.21.15.98:0/1014315590 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2413e970 con 0x7f4b2413ee90 2024-03-29T10:52:42.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b21d9b640 10 monclient: _finish_auth 0 2024-03-29T10:52:42.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b21d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:52:42.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b21d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:42.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b21d9b640 1 -- 172.21.15.98:0/1014315590 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4b1c001a10 con 0x7f4b2413ee90 2024-03-29T10:52:42.711 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b21d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b21d9b640 10 monclient: dump: 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b29220640 5 monclient: authenticate success, global_id 4550 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.726+0000 7f4b29220640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:42.712 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.727+0000 7f4b21d9b640 1 -- 172.21.15.98:0/1014315590 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b1c001be0 con 0x7f4b2413ee90 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.727+0000 7f4b21d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.727+0000 7f4b21d9b640 1 -- 172.21.15.98:0/1014315590 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4b1c002030 con 0x7f4b2413ee90 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.727+0000 7f4b29220640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.727+0000 7f4b29220640 4 set_mon_vals no callback set 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.727+0000 7f4b21d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.727+0000 7f4b21d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.727+0000 7f4b21d9b640 10 monclient: dump: 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:42.713 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:42.714 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:42.714 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:42.714 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:42.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.728+0000 7f4b29220640 10 monclient: shutdown 2024-03-29T10:52:42.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.728+0000 7f4b29220640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:42.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.728+0000 7f4b29220640 1 -- 172.21.15.98:0/1014315590 >> v1:172.21.15.98:6790/0 conn(0x7f4b2413ee90 legacy=0x7f4b24146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:42.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.728+0000 7f4b29220640 1 -- 172.21.15.98:0/1014315590 shutdown_connections 2024-03-29T10:52:42.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.728+0000 7f4b29220640 1 -- 172.21.15.98:0/1014315590 >> 172.21.15.98:0/1014315590 conn(0x7f4b240b35e0 msgr2=0x7f4b240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:42.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.728+0000 7f4b29220640 1 -- 172.21.15.98:0/1014315590 shutdown_connections 2024-03-29T10:52:42.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.728+0000 7f4b29220640 1 -- 172.21.15.98:0/1014315590 wait complete. 2024-03-29T10:52:42.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.729+0000 7f4b29220640 10 monclient: build_initial_monmap 2024-03-29T10:52:42.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.729+0000 7f4b29220640 10 monclient: monmap: 2024-03-29T10:52:42.715 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.730+0000 7f4b29220640 1 librados: starting msgr at 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.730+0000 7f4b29220640 1 librados: starting objecter 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.730+0000 7f4b29220640 1 Processor -- start 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.730+0000 7f4b29220640 1 -- start start 2024-03-29T10:52:42.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.730+0000 7f4b29220640 1 librados: setting wanted keys 2024-03-29T10:52:42.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.730+0000 7f4b29220640 1 librados: calling monclient init 2024-03-29T10:52:42.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.730+0000 7f4b29220640 10 monclient: init 2024-03-29T10:52:42.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.731+0000 7f4b29220640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:42.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.731+0000 7f4b29220640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:42.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.731+0000 7f4b29220640 10 monclient(hunting): picked mon.noname-c con 0x7f4b240a9ac0 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:42.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.731+0000 7f4b29220640 10 monclient(hunting): picked mon.noname-a con 0x7f4b240b4100 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:42.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.734+0000 7f4b2259c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4b240b4100 0x7f4b241d3080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53141/0 (socket says 172.21.15.98:53141) 2024-03-29T10:52:42.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.735+0000 7f4b22d9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f4b240a9ac0 0x7f4b24071b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:54106/0 (socket says 172.21.15.98:54106) 2024-03-29T10:52:42.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.735+0000 7f4b29220640 10 monclient(hunting): picked mon.noname-b con 0x7f4b2413ee90 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:42.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.735+0000 7f4b2259c640 1 -- 172.21.15.98:0/3603873949 learned_addr learned my addr 172.21.15.98:0/3603873949 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:42.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.735+0000 7f4b0b7fe640 10 client.?.objecter ms_handle_connect 0x7f4b240b4100 2024-03-29T10:52:42.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.735+0000 7f4b0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.735+0000 7f4b0b7fe640 10 client.?.objecter ms_handle_connect 0x7f4b2413ee90 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.735+0000 7f4b0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.735+0000 7f4b0b7fe640 10 client.?.objecter ms_handle_connect 0x7f4b240a9ac0 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.735+0000 7f4b0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b29220640 1 -- 172.21.15.98:0/3603873949 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b24140a10 con 0x7f4b240b4100 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b29220640 1 -- 172.21.15.98:0/3603873949 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b24141db0 con 0x7f4b2413ee90 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b29220640 1 -- 172.21.15.98:0/3603873949 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b240722a0 con 0x7f4b240a9ac0 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452909046 0 0) 0x7f4b24140a10 con 0x7f4b240b4100 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b29220640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b29220640 10 monclient(hunting): authenticate will time out at 1382.934607s 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:42.726 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b0b7fe640 10 monclient(hunting): my global_id is 4500 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b10004eb0 con 0x7f4b240b4100 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665586483 0 0) 0x7f4b24141db0 con 0x7f4b2413ee90 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b0b7fe640 10 monclient(hunting): my global_id is 4513 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.736+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b24140a10 con 0x7f4b2413ee90 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.737+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958953682 0 0) 0x7f4b240722a0 con 0x7f4b240a9ac0 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.737+0000 7f4b0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.737+0000 7f4b0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.737+0000 7f4b0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.737+0000 7f4b0b7fe640 10 monclient(hunting): my global_id is 4553 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.737+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b24141db0 con 0x7f4b240a9ac0 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.737+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1691723885 0 0) 0x7f4b24141db0 con 0x7f4b240a9ac0 2024-03-29T10:52:42.727 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b240722a0 con 0x7f4b240a9ac0 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4b180017e0 con 0x7f4b240a9ac0 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 10 monclient(hunting): dump: 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:42.728 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1381084926 0 0) 0x7f4b240722a0 con 0x7f4b240a9ac0 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 >> v1:172.21.15.189:6789/0 conn(0x7f4b2413ee90 legacy=0x7f4b241d3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 >> v1:172.21.15.98:6789/0 conn(0x7f4b240b4100 legacy=0x7f4b241d3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 1 monclient: found mon.c 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b24072430 con 0x7f4b240a9ac0 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.738+0000 7f4b0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b29220640 5 monclient: authenticate success, global_id 4553 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b29220640 10 monclient: _renew_subs 2024-03-29T10:52:42.729 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b29220640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:42.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b29220640 1 -- 172.21.15.98:0/3603873949 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b2413f9c0 con 0x7f4b240a9ac0 2024-03-29T10:52:42.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b29220640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:42.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b29220640 10 monclient: _renew_subs 2024-03-29T10:52:42.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b29220640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:42.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b29220640 1 -- 172.21.15.98:0/3603873949 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b2413fef0 con 0x7f4b240a9ac0 2024-03-29T10:52:42.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b29220640 1 librados: init done 2024-03-29T10:52:42.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b18001cf0 con 0x7f4b240a9ac0 2024-03-29T10:52:42.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:42.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4b18003790 con 0x7f4b240a9ac0 2024-03-29T10:52:42.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:42.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:42.730 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b0b7fe640 10 monclient: dump: 2024-03-29T10:52:42.730 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.739+0000 7f4b20d99640 4 set_mon_vals no callback set 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.740+0000 7f4b29220640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.740+0000 7f4b29220640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.740+0000 7f4b29220640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.740+0000 7f4b29220640 1 -- 172.21.15.98:0/3603873949 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b04005080 con 0x7f4b240a9ac0 2024-03-29T10:52:42.731 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.741+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f4b180038f0 con 0x7f4b240a9ac0 2024-03-29T10:52:42.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.742+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f4b180546a0 con 0x7f4b240a9ac0 2024-03-29T10:52:42.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.742+0000 7f4b0b7fe640 10 client.4553.objecter ms_dispatch 0x7f4b24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:42.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.742+0000 7f4b0b7fe640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:42.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.742+0000 7f4b0b7fe640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:42.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.742+0000 7f4b0b7fe640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-29T10:52:42.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.742+0000 7f4b2259c640 10 monclient: get_auth_request con 0x7f4b1006ddd0 auth_method 0 2024-03-29T10:52:42.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.743+0000 7f4b0b7fe640 10 client.4553.objecter ms_handle_connect 0x7f4b1006ddd0 2024-03-29T10:52:42.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.751+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b18050940 con 0x7f4b240a9ac0 2024-03-29T10:52:42.733 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.751+0000 7f4b0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:42.734 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:42.751+0000 7f4b0b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:43.065 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.083+0000 7f4b29220640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:43.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.083+0000 7f4b29220640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:43.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.083+0000 7f4b29220640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:43.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.083+0000 7f4b29220640 1 -- 172.21.15.98:0/3603873949 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b04005330 con 0x7f4b240a9ac0 2024-03-29T10:52:43.066 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:52:43.066 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.084+0000 7f4b0b7fe640 1 -- 172.21.15.98:0/3603873949 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f4b18050ae0 con 0x7f4b240a9ac0 2024-03-29T10:52:43.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.084+0000 7f4b0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:43.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.084+0000 7f4b0b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:43.068 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:43.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.088+0000 7f4b29220640 10 librados: watch_flush enter 2024-03-29T10:52:43.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.088+0000 7f4b29220640 10 librados: watch_flush exit 2024-03-29T10:52:43.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.088+0000 7f4b29220640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-29T10:52:43.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.088+0000 7f4b29220640 10 client.4553.objecter successfully canceled tick 2024-03-29T10:52:43.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.088+0000 7f4b29220640 1 -- 172.21.15.98:0/3603873949 >> v1:172.21.15.98:6816/1481799426 conn(0x7f4b1006ddd0 legacy=0x7f4b10070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:43.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.088+0000 7f4b29220640 10 monclient: shutdown 2024-03-29T10:52:43.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.088+0000 7f4b29220640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:43.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.088+0000 7f4b29220640 1 -- 172.21.15.98:0/3603873949 >> v1:172.21.15.98:6790/0 conn(0x7f4b240a9ac0 legacy=0x7f4b24071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:43.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.088+0000 7f4b29220640 1 -- 172.21.15.98:0/3603873949 shutdown_connections 2024-03-29T10:52:43.071 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.088+0000 7f4b29220640 1 -- 172.21.15.98:0/3603873949 >> 172.21.15.98:0/3603873949 conn(0x7f4b240b35e0 msgr2=0x7f4b241367a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:43.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.090+0000 7f4b29220640 1 -- 172.21.15.98:0/3603873949 shutdown_connections 2024-03-29T10:52:43.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.090+0000 7f4b29220640 1 -- 172.21.15.98:0/3603873949 wait complete. 2024-03-29T10:52:43.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:43.090+0000 7f4b29220640 1 librados: shutdown 2024-03-29T10:52:43.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:49.901 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:50.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.076+0000 7f3595463640 10 monclient: get_monmap_and_config 2024-03-29T10:52:50.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.076+0000 7f3595463640 10 monclient: build_initial_monmap 2024-03-29T10:52:50.062 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.076+0000 7f3595463640 10 monclient: monmap: 2024-03-29T10:52:50.062 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:50.063 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:50.063 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:52:50.078261+0000 2024-03-29T10:52:50.063 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:52:50.078261+0000 2024-03-29T10:52:50.063 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:50.063 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:50.063 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:50.063 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:52:50.064 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:52:50.064 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:50.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.078+0000 7f3595463640 1 Processor -- start 2024-03-29T10:52:50.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.078+0000 7f3595463640 1 -- start start 2024-03-29T10:52:50.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.078+0000 7f3595463640 10 monclient: init 2024-03-29T10:52:50.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.079+0000 7f3595463640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:50.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.079+0000 7f3595463640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:52:50.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.079+0000 7f3595463640 10 monclient(hunting): picked mon.noname-c con 0x7f35880389e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:50.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.079+0000 7f3595463640 10 monclient(hunting): picked mon.noname-a con 0x7f3588038dc0 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:50.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.079+0000 7f3595463640 10 monclient(hunting): picked mon.noname-b con 0x7f3588030660 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:50.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.079+0000 7f3595463640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35880c54c0 con 0x7f3588038dc0 2024-03-29T10:52:50.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.079+0000 7f3595463640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3588036430 con 0x7f35880389e0 2024-03-29T10:52:50.067 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.079+0000 7f3595463640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3588039840 con 0x7f3588030660 2024-03-29T10:52:50.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.079+0000 7f3595463640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:50.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.079+0000 7f3595463640 10 monclient(hunting): authenticate will time out at 1390.277186s 2024-03-29T10:52:50.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.081+0000 7f3594c62640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f3588030660 0x7f35880d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:50044/0 (socket says 172.21.15.98:50044) 2024-03-29T10:52:50.068 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.081+0000 7f3594c62640 1 -- 172.21.15.98:0/376743018 learned_addr learned my addr 172.21.15.98:0/376743018 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:50.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.081+0000 7f358effd640 1 -- 172.21.15.98:0/376743018 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540542179 0 0) 0x7f3588039840 con 0x7f3588030660 2024-03-29T10:52:50.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.081+0000 7f358effd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:50.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.081+0000 7f358effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:50.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.081+0000 7f358effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:50.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.081+0000 7f358effd640 10 monclient(hunting): my global_id is 4556 2024-03-29T10:52:50.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.081+0000 7f358effd640 1 -- 172.21.15.98:0/376743018 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3580003520 con 0x7f3588030660 2024-03-29T10:52:50.069 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.082+0000 7f358effd640 1 -- 172.21.15.98:0/376743018 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3263049620 0 0) 0x7f3580003520 con 0x7f3588030660 2024-03-29T10:52:50.070 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.082+0000 7f358effd640 1 -- 172.21.15.98:0/376743018 >> v1:172.21.15.189:6789/0 conn(0x7f35880389e0 legacy=0x7f3588033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:52:50.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f358effd640 1 -- 172.21.15.98:0/376743018 >> v1:172.21.15.98:6789/0 conn(0x7f3588038dc0 legacy=0x7f35880cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:50.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f358effd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:50.072 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f358effd640 1 monclient: found mon.noname-b 2024-03-29T10:52:50.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f358effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:50.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f358effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:52:50.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f358effd640 1 -- 172.21.15.98:0/376743018 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3588035690 con 0x7f3588030660 2024-03-29T10:52:50.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f358effd640 10 monclient: _finish_auth 0 2024-03-29T10:52:50.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f358effd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:50.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f358effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:50.073 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f358effd640 1 -- 172.21.15.98:0/376743018 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f359004f730 con 0x7f3588030660 2024-03-29T10:52:50.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f358effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:50.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f358effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:50.074 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f358effd640 10 monclient: dump: 2024-03-29T10:52:50.074 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:50.074 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:50.074 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:50.074 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:50.074 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:50.075 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:50.075 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:50.075 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:50.075 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:50.075 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:50.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f3595463640 5 monclient: authenticate success, global_id 4556 2024-03-29T10:52:50.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.083+0000 7f3595463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:50.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.084+0000 7f358effd640 1 -- 172.21.15.98:0/376743018 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3590075f10 con 0x7f3588030660 2024-03-29T10:52:50.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.084+0000 7f358effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:50.075 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.084+0000 7f358effd640 1 -- 172.21.15.98:0/376743018 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f35900758b0 con 0x7f3588030660 2024-03-29T10:52:50.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.084+0000 7f3595463640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:50.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.084+0000 7f3595463640 4 set_mon_vals no callback set 2024-03-29T10:52:50.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.085+0000 7f3595463640 10 monclient: shutdown 2024-03-29T10:52:50.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.085+0000 7f3595463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:50.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.085+0000 7f3595463640 1 -- 172.21.15.98:0/376743018 >> v1:172.21.15.98:6790/0 conn(0x7f3588030660 legacy=0x7f35880d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:50.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.085+0000 7f358effd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:52:50.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.086+0000 7f3595463640 1 -- 172.21.15.98:0/376743018 shutdown_connections 2024-03-29T10:52:50.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.086+0000 7f3595463640 1 -- 172.21.15.98:0/376743018 >> 172.21.15.98:0/376743018 conn(0x7f3588005c00 msgr2=0x7f3588006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:50.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.086+0000 7f3595463640 1 -- 172.21.15.98:0/376743018 shutdown_connections 2024-03-29T10:52:50.076 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.086+0000 7f3595463640 1 -- 172.21.15.98:0/376743018 wait complete. 2024-03-29T10:52:50.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.087+0000 7f3595463640 10 monclient: build_initial_monmap 2024-03-29T10:52:50.077 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.087+0000 7f3595463640 10 monclient: monmap: 2024-03-29T10:52:50.077 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:50.077 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:50.077 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:52:50.077 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:52:50.077 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:50.077 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:50.077 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:50.078 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:52:50.078 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:52:50.078 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:50.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.087+0000 7f3595463640 1 librados: starting msgr at 2024-03-29T10:52:50.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.087+0000 7f3595463640 1 librados: starting objecter 2024-03-29T10:52:50.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.088+0000 7f3595463640 1 Processor -- start 2024-03-29T10:52:50.078 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.088+0000 7f3595463640 1 -- start start 2024-03-29T10:52:50.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.088+0000 7f3595463640 1 librados: setting wanted keys 2024-03-29T10:52:50.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.088+0000 7f3595463640 1 librados: calling monclient init 2024-03-29T10:52:50.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.088+0000 7f3595463640 10 monclient: init 2024-03-29T10:52:50.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.089+0000 7f3595463640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:50.079 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.089+0000 7f3595463640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:52:50.080 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.089+0000 7f3595463640 10 monclient(hunting): picked mon.noname-a con 0x7f3588030660 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:50.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.090+0000 7f3595463640 10 monclient(hunting): picked mon.noname-b con 0x7f35880389e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:50.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.090+0000 7f3595463640 10 monclient(hunting): picked mon.noname-c con 0x7f3588009fe0 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:50.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.090+0000 7f3595463640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35880becd0 con 0x7f3588030660 2024-03-29T10:52:50.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.090+0000 7f3595463640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358815b890 con 0x7f35880389e0 2024-03-29T10:52:50.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.090+0000 7f3595463640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358800b630 con 0x7f3588009fe0 2024-03-29T10:52:50.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.090+0000 7f3595463640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:50.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.090+0000 7f3595463640 10 monclient(hunting): authenticate will time out at 1390.288468s 2024-03-29T10:52:50.093 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.091+0000 7f3594c62640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f3588009fe0 0x7f35880be5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:58044/0 (socket says 172.21.15.98:58044) 2024-03-29T10:52:50.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.091+0000 7f3594c62640 1 -- 172.21.15.98:0/3969942784 learned_addr learned my addr 172.21.15.98:0/3969942784 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:50.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 client.?.objecter ms_handle_connect 0x7f3588009fe0 2024-03-29T10:52:50.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:50.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 client.?.objecter ms_handle_connect 0x7f35880389e0 2024-03-29T10:52:50.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:50.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 client.?.objecter ms_handle_connect 0x7f3588030660 2024-03-29T10:52:50.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:50.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615440748 0 0) 0x7f358800b630 con 0x7f3588009fe0 2024-03-29T10:52:50.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:50.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:50.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:50.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): my global_id is 4559 2024-03-29T10:52:50.094 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3574004eb0 con 0x7f3588009fe0 2024-03-29T10:52:50.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769863806 0 0) 0x7f35880becd0 con 0x7f3588030660 2024-03-29T10:52:50.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:50.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:50.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:50.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): my global_id is 4506 2024-03-29T10:52:50.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358800b630 con 0x7f3588030660 2024-03-29T10:52:50.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166782938 0 0) 0x7f358815b890 con 0x7f35880389e0 2024-03-29T10:52:50.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:50.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:50.095 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:50.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): my global_id is 4516 2024-03-29T10:52:50.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35880becd0 con 0x7f35880389e0 2024-03-29T10:52:50.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 233561883 0 0) 0x7f3574004eb0 con 0x7f3588009fe0 2024-03-29T10:52:50.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f358815b890 con 0x7f3588009fe0 2024-03-29T10:52:50.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f359004f400 con 0x7f3588009fe0 2024-03-29T10:52:50.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:50.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:52:50.096 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.095+0000 7f358cff9640 10 monclient(hunting): dump: 2024-03-29T10:52:50.097 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:50.097 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:50.097 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:50.097 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:50.097 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:50.097 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:50.097 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:50.097 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:50.097 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:50.098 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:50.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:50.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2733793928 0 0) 0x7f358800b630 con 0x7f3588030660 2024-03-29T10:52:50.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3574004eb0 con 0x7f3588030660 2024-03-29T10:52:50.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f357c001720 con 0x7f3588030660 2024-03-29T10:52:50.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:50.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:50.098 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 10 monclient(hunting): dump: 2024-03-29T10:52:50.098 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:50.098 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:50.098 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:50.099 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:50.099 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:50.099 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:50.099 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:50.099 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:50.099 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:50.099 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:50.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 119076830 0 0) 0x7f358815b890 con 0x7f3588009fe0 2024-03-29T10:52:50.099 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 >> v1:172.21.15.189:6789/0 conn(0x7f35880389e0 legacy=0x7f35880bdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:50.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 >> v1:172.21.15.98:6789/0 conn(0x7f3588030660 legacy=0x7f358815c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:50.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:52:50.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 1 monclient: found mon.c 2024-03-29T10:52:50.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:50.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:50.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358815bb30 con 0x7f3588009fe0 2024-03-29T10:52:50.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 10 monclient: _finish_auth 0 2024-03-29T10:52:50.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:52:50.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f358cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:50.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f3595463640 5 monclient: authenticate success, global_id 4559 2024-03-29T10:52:50.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f3595463640 10 monclient: _renew_subs 2024-03-29T10:52:50.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f3595463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:50.100 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f3595463640 1 -- 172.21.15.98:0/3969942784 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358815c760 con 0x7f3588009fe0 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f3595463640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f3595463640 10 monclient: _renew_subs 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f3595463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f3595463640 1 -- 172.21.15.98:0/3969942784 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f358802d1c0 con 0x7f3588009fe0 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.096+0000 7f3595463640 1 librados: init done 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.097+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f359004fdd0 con 0x7f3588009fe0 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.097+0000 7f358cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.097+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3590067bf0 con 0x7f3588009fe0 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.097+0000 7f358cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.097+0000 7f358cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.097+0000 7f358cff9640 10 monclient: dump: 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:50.101 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:50.102 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:50.102 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:50.102 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:50.102 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:50.102 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:50.102 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:50.103 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:50.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.097+0000 7f358e7fc640 4 set_mon_vals no callback set 2024-03-29T10:52:50.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.098+0000 7f3595463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:50.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.098+0000 7f3595463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:50.103 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.098+0000 7f3595463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:50.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.098+0000 7f3595463640 1 -- 172.21.15.98:0/3969942784 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3570005080 con 0x7f3588009fe0 2024-03-29T10:52:50.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.100+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f3590067d50 con 0x7f3588009fe0 2024-03-29T10:52:50.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.101+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f35900ff4d0 con 0x7f3588009fe0 2024-03-29T10:52:50.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.101+0000 7f358cff9640 10 client.4559.objecter ms_dispatch 0x7f3590000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:50.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.101+0000 7f358cff9640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:50.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.101+0000 7f358cff9640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:50.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.101+0000 7f358ffff640 10 monclient: get_auth_request con 0x7f357406dff0 auth_method 0 2024-03-29T10:52:50.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.101+0000 7f358cff9640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T10:52:50.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.101+0000 7f358cff9640 10 client.4559.objecter ms_handle_connect 0x7f357406dff0 2024-03-29T10:52:50.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.109+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35900cadb0 con 0x7f3588009fe0 2024-03-29T10:52:50.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.109+0000 7f358cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:50.104 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.109+0000 7f358cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:50.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.437+0000 7f3595463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:50.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.437+0000 7f3595463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:50.419 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.437+0000 7f3595463640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:52:50.420 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.437+0000 7f3595463640 1 -- 172.21.15.98:0/3969942784 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3570005330 con 0x7f3588009fe0 2024-03-29T10:52:50.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.439+0000 7f358cff9640 1 -- 172.21.15.98:0/3969942784 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f35900ccce0 con 0x7f3588009fe0 2024-03-29T10:52:50.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.439+0000 7f358cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:50.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.439+0000 7f358cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:50.423 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:52:50.423 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:50.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.444+0000 7f3595463640 10 librados: watch_flush enter 2024-03-29T10:52:50.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.444+0000 7f3595463640 10 librados: watch_flush exit 2024-03-29T10:52:50.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.444+0000 7f3595463640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T10:52:50.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.444+0000 7f3595463640 10 client.4559.objecter successfully canceled tick 2024-03-29T10:52:50.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.444+0000 7f3595463640 1 -- 172.21.15.98:0/3969942784 >> v1:172.21.15.98:6816/1481799426 conn(0x7f357406dff0 legacy=0x7f35740704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:50.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.444+0000 7f3595463640 10 monclient: shutdown 2024-03-29T10:52:50.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.444+0000 7f3595463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:50.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.444+0000 7f3595463640 1 -- 172.21.15.98:0/3969942784 >> v1:172.21.15.98:6790/0 conn(0x7f3588009fe0 legacy=0x7f35880be5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:50.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.444+0000 7f3595463640 1 -- 172.21.15.98:0/3969942784 shutdown_connections 2024-03-29T10:52:50.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.444+0000 7f3595463640 1 -- 172.21.15.98:0/3969942784 >> 172.21.15.98:0/3969942784 conn(0x7f3588005c00 msgr2=0x7f35880c26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:50.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.445+0000 7f3595463640 1 -- 172.21.15.98:0/3969942784 shutdown_connections 2024-03-29T10:52:50.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.445+0000 7f3595463640 1 -- 172.21.15.98:0/3969942784 wait complete. 2024-03-29T10:52:50.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:50.445+0000 7f3595463640 1 librados: shutdown 2024-03-29T10:52:50.446 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:52:57.253 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:52:57.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.429+0000 7fe5ae263640 10 monclient: get_monmap_and_config 2024-03-29T10:52:57.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.429+0000 7fe5ae263640 10 monclient: build_initial_monmap 2024-03-29T10:52:57.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.429+0000 7fe5ae263640 10 monclient: monmap: 2024-03-29T10:52:57.412 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:57.412 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:57.412 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:52:57.431480+0000 2024-03-29T10:52:57.412 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:52:57.431480+0000 2024-03-29T10:52:57.412 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:57.412 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:57.413 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:57.413 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:52:57.413 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:52:57.413 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:57.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.432+0000 7fe5ae263640 1 Processor -- start 2024-03-29T10:52:57.415 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.432+0000 7fe5ae263640 1 -- start start 2024-03-29T10:52:57.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.432+0000 7fe5ae263640 10 monclient: init 2024-03-29T10:52:57.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.433+0000 7fe5ae263640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:57.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.433+0000 7fe5ae263640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:52:57.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.433+0000 7fe5ae263640 10 monclient(hunting): picked mon.noname-b con 0x7fe5a80a9e70 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:57.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.433+0000 7fe5ae263640 10 monclient(hunting): picked mon.noname-c con 0x7fe5a81420b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:57.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.434+0000 7fe5ae263640 10 monclient(hunting): picked mon.noname-a con 0x7fe5a80ae0e0 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:57.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.434+0000 7fe5ae263640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a80ab840 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.416 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.434+0000 7fe5ae263640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a80abab0 con 0x7fe5a81420b0 2024-03-29T10:52:57.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.434+0000 7fe5ae263640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a8141b90 con 0x7fe5a80a9e70 2024-03-29T10:52:57.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.434+0000 7fe5ae263640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:57.417 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.434+0000 7fe5ae263640 10 monclient(hunting): authenticate will time out at 1397.632183s 2024-03-29T10:52:57.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.437+0000 7fe5a77fe640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe5a81420b0 0x7fe5a80af1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:51355/0 (socket says 172.21.15.98:51355) 2024-03-29T10:52:57.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.437+0000 7fe5a77fe640 1 -- 172.21.15.98:0/346949566 learned_addr learned my addr 172.21.15.98:0/346949566 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:57.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.437+0000 7fe5a6ffd640 1 -- 172.21.15.98:0/346949566 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631818124 0 0) 0x7fe5a8141b90 con 0x7fe5a80a9e70 2024-03-29T10:52:57.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.437+0000 7fe5a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:57.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.437+0000 7fe5a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:57.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.437+0000 7fe5a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:57.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.437+0000 7fe5a6ffd640 10 monclient(hunting): my global_id is 4562 2024-03-29T10:52:57.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.437+0000 7fe5a6ffd640 1 -- 172.21.15.98:0/346949566 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe598003520 con 0x7fe5a80a9e70 2024-03-29T10:52:57.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 1 -- 172.21.15.98:0/346949566 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539023599 0 0) 0x7fe5a80abab0 con 0x7fe5a81420b0 2024-03-29T10:52:57.421 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:52:57.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:57.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:57.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 10 monclient(hunting): my global_id is 4519 2024-03-29T10:52:57.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 1 -- 172.21.15.98:0/346949566 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a8141b90 con 0x7fe5a81420b0 2024-03-29T10:52:57.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 1 -- 172.21.15.98:0/346949566 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1136596762 0 0) 0x7fe598003520 con 0x7fe5a80a9e70 2024-03-29T10:52:57.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 1 -- 172.21.15.98:0/346949566 >> v1:172.21.15.189:6789/0 conn(0x7fe5a81420b0 legacy=0x7fe5a80af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:57.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 1 -- 172.21.15.98:0/346949566 >> v1:172.21.15.98:6789/0 conn(0x7fe5a80ae0e0 legacy=0x7fe5a80acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:52:57.422 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:52:57.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 1 monclient: found mon.noname-b 2024-03-29T10:52:57.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:57.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:52:57.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 1 -- 172.21.15.98:0/346949566 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a8141850 con 0x7fe5a80a9e70 2024-03-29T10:52:57.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:52:57.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:52:57.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:57.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 1 -- 172.21.15.98:0/346949566 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe59c001690 con 0x7fe5a80a9e70 2024-03-29T10:52:57.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:57.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:57.423 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5a6ffd640 10 monclient: dump: 2024-03-29T10:52:57.423 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:57.424 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:57.424 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:57.424 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:57.424 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:57.424 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:57.424 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:57.424 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:57.424 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:57.424 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:57.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5ae263640 5 monclient: authenticate success, global_id 4562 2024-03-29T10:52:57.424 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.438+0000 7fe5ae263640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:52:57.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.439+0000 7fe5a6ffd640 1 -- 172.21.15.98:0/346949566 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe59c0017f0 con 0x7fe5a80a9e70 2024-03-29T10:52:57.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.439+0000 7fe5a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:57.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.439+0000 7fe5a6ffd640 1 -- 172.21.15.98:0/346949566 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe59c001c00 con 0x7fe5a80a9e70 2024-03-29T10:52:57.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.439+0000 7fe5a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:57.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.439+0000 7fe5a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:57.427 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.439+0000 7fe5a6ffd640 10 monclient: dump: 2024-03-29T10:52:57.427 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:57.427 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:57.427 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:57.427 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:57.428 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:57.428 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:57.428 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:57.428 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:57.428 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:57.428 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:57.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.439+0000 7fe5ae263640 10 monclient: get_monmap_and_config success 2024-03-29T10:52:57.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.439+0000 7fe5ae263640 4 set_mon_vals no callback set 2024-03-29T10:52:57.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.440+0000 7fe5ae263640 10 monclient: shutdown 2024-03-29T10:52:57.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.440+0000 7fe5ae263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:57.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.440+0000 7fe5ae263640 1 -- 172.21.15.98:0/346949566 >> v1:172.21.15.98:6790/0 conn(0x7fe5a80a9e70 legacy=0x7fe5a80aa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:57.428 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.440+0000 7fe5ae263640 1 -- 172.21.15.98:0/346949566 shutdown_connections 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.440+0000 7fe5ae263640 1 -- 172.21.15.98:0/346949566 >> 172.21.15.98:0/346949566 conn(0x7fe5a80b36e0 msgr2=0x7fe5a80b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.440+0000 7fe5ae263640 1 -- 172.21.15.98:0/346949566 shutdown_connections 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.440+0000 7fe5ae263640 1 -- 172.21.15.98:0/346949566 wait complete. 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.443+0000 7fe5ae263640 10 monclient: build_initial_monmap 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.443+0000 7fe5ae263640 10 monclient: monmap: 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:52:57.429 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:52:57.430 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:57.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.443+0000 7fe5ae263640 1 librados: starting msgr at 2024-03-29T10:52:57.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.443+0000 7fe5ae263640 1 librados: starting objecter 2024-03-29T10:52:57.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.443+0000 7fe5ae263640 1 Processor -- start 2024-03-29T10:52:57.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.444+0000 7fe5ae263640 1 -- start start 2024-03-29T10:52:57.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.444+0000 7fe5ae263640 1 librados: setting wanted keys 2024-03-29T10:52:57.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.444+0000 7fe5ae263640 1 librados: calling monclient init 2024-03-29T10:52:57.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.444+0000 7fe5ae263640 10 monclient: init 2024-03-29T10:52:57.430 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.444+0000 7fe5ae263640 10 monclient: _reopen_session rank -1 2024-03-29T10:52:57.431 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.444+0000 7fe5ae263640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:52:57.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.447+0000 7fe5ae263640 10 monclient(hunting): picked mon.noname-c con 0x7fe5a80a9e70 addr v1:172.21.15.98:6790/0 2024-03-29T10:52:57.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.447+0000 7fe5ae263640 10 monclient(hunting): picked mon.noname-a con 0x7fe5a80ae0e0 addr v1:172.21.15.98:6789/0 2024-03-29T10:52:57.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.447+0000 7fe5ae263640 10 monclient(hunting): picked mon.noname-b con 0x7fe5a81420b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:52:57.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.447+0000 7fe5ae263640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a8127ec0 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.447+0000 7fe5ae263640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a81281a0 con 0x7fe5a81420b0 2024-03-29T10:52:57.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.450+0000 7fe5ad001640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe5a81420b0 0x7fe5a806fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:51405/0 (socket says 172.21.15.98:51405) 2024-03-29T10:52:57.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.450+0000 7fe5ad001640 1 -- 172.21.15.98:0/1294004181 learned_addr learned my addr 172.21.15.98:0/1294004181 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:52:57.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.451+0000 7fe5ae263640 1 -- 172.21.15.98:0/1294004181 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a8128480 con 0x7fe5a80a9e70 2024-03-29T10:52:57.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.451+0000 7fe5ae263640 10 monclient(hunting): _renew_subs 2024-03-29T10:52:57.434 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.451+0000 7fe5ae263640 10 monclient(hunting): authenticate will time out at 1397.649444s 2024-03-29T10:52:57.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 10 client.?.objecter ms_handle_connect 0x7fe5a81420b0 2024-03-29T10:52:57.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:57.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 10 client.?.objecter ms_handle_connect 0x7fe5a80a9e70 2024-03-29T10:52:57.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:57.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398583392 0 0) 0x7fe5a81281a0 con 0x7fe5a81420b0 2024-03-29T10:52:57.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:57.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:57.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:57.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 10 monclient(hunting): my global_id is 4522 2024-03-29T10:52:57.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe58c004eb0 con 0x7fe5a81420b0 2024-03-29T10:52:57.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640932557 0 0) 0x7fe5a8128480 con 0x7fe5a80a9e70 2024-03-29T10:52:57.435 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:57.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:57.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:57.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 10 monclient(hunting): my global_id is 4565 2024-03-29T10:52:57.436 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.452+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a81281a0 con 0x7fe5a80a9e70 2024-03-29T10:52:57.440 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.453+0000 7fe5a4ff9640 10 client.?.objecter ms_handle_connect 0x7fe5a80ae0e0 2024-03-29T10:52:57.440 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.453+0000 7fe5a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:52:57.440 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771035715 0 0) 0x7fe5a8127ec0 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.440 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:52:57.440 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:52:57.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:52:57.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 10 monclient(hunting): my global_id is 4509 2024-03-29T10:52:57.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a8128480 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3658526811 0 0) 0x7fe58c004eb0 con 0x7fe5a81420b0 2024-03-29T10:52:57.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe58c0066d0 con 0x7fe5a81420b0 2024-03-29T10:52:57.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe5a0001c90 con 0x7fe5a81420b0 2024-03-29T10:52:57.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:57.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:52:57.441 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 10 monclient(hunting): dump: 2024-03-29T10:52:57.441 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:57.441 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:57.441 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:57.441 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 365686633 0 0) 0x7fe5a81281a0 con 0x7fe5a80a9e70 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.454+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe58c004eb0 con 0x7fe5a80a9e70 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.455+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe59c001b00 con 0x7fe5a80a9e70 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.455+0000 7fe5a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.455+0000 7fe5a4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.455+0000 7fe5a4ff9640 10 monclient(hunting): dump: 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:57.442 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:57.443 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:57.443 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:57.443 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:57.443 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:57.443 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:57.443 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:57.443 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:57.443 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:57.443 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.455+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2398557596 0 0) 0x7fe5a8128480 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.443 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.455+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5a81281a0 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.455+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe594001790 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.455+0000 7fe5a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.455+0000 7fe5a4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.455+0000 7fe5a4ff9640 10 monclient(hunting): dump: 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:57.444 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 452685612 0 0) 0x7fe5a81281a0 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 >> v1:172.21.15.98:6790/0 conn(0x7fe5a80a9e70 legacy=0x7fe5a814e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 >> v1:172.21.15.189:6789/0 conn(0x7fe5a81420b0 legacy=0x7fe5a806fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5a4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5a4ff9640 1 monclient: found mon.a 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5a4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a806ee50 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5a4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5a4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5ae263640 5 monclient: authenticate success, global_id 4509 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5ae263640 10 monclient: _renew_subs 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5ae263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:52:57.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5ae263640 1 -- 172.21.15.98:0/1294004181 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a81266a0 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5ae263640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:52:57.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5ae263640 10 monclient: _renew_subs 2024-03-29T10:52:57.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5ae263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:52:57.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5ae263640 1 -- 172.21.15.98:0/1294004181 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a80718e0 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.456+0000 7fe5ae263640 1 librados: init done 2024-03-29T10:52:57.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.459+0000 7fe5ae263640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:57.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.459+0000 7fe5ae263640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:57.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.459+0000 7fe5ae263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:52:57.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.459+0000 7fe5ae263640 1 -- 172.21.15.98:0/1294004181 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a806f090 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.461+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe594001c60 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.461+0000 7fe5a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:52:57.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.461+0000 7fe5a5ffb640 4 set_mon_vals no callback set 2024-03-29T10:52:57.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.461+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fe594001790 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.461+0000 7fe5a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:52:57.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.461+0000 7fe5a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:52:57.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.461+0000 7fe5a4ff9640 10 monclient: dump: 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.461+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fe59401a690 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.462+0000 7fe5a7fff640 10 monclient: get_auth_request con 0x7fe58c06e640 auth_method 0 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.462+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fe5940845d0 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.462+0000 7fe5a4ff9640 10 client.4509.objecter ms_dispatch 0x7fe5a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:52:57.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.462+0000 7fe5a4ff9640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:52:57.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.462+0000 7fe5a4ff9640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:52:57.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.462+0000 7fe5a4ff9640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-29T10:52:57.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.463+0000 7fe5a4ff9640 10 client.4509.objecter ms_handle_connect 0x7fe58c06e640 2024-03-29T10:52:57.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.475+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5a806f090 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.475+0000 7fe5a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:52:57.457 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.475+0000 7fe5a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:52:57.803 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.821+0000 7fe5ae263640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:57.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.821+0000 7fe5ae263640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:57.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.821+0000 7fe5ae263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:52:57.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.821+0000 7fe5ae263640 1 -- 172.21.15.98:0/1294004181 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5a8127c70 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.822+0000 7fe5a4ff9640 1 -- 172.21.15.98:0/1294004181 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fe59404fd60 con 0x7fe5a80ae0e0 2024-03-29T10:52:57.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.822+0000 7fe5a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:52:57.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.822+0000 7fe5a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:52:57.805 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:52:57.805 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:52:57.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.826+0000 7fe5ae263640 10 librados: watch_flush enter 2024-03-29T10:52:57.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.826+0000 7fe5ae263640 10 librados: watch_flush exit 2024-03-29T10:52:57.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.826+0000 7fe5ae263640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-29T10:52:57.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.826+0000 7fe5ae263640 10 client.4509.objecter successfully canceled tick 2024-03-29T10:52:57.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.826+0000 7fe5ae263640 1 -- 172.21.15.98:0/1294004181 >> v1:172.21.15.98:6816/1481799426 conn(0x7fe58c06e640 legacy=0x7fe58c070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:57.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.826+0000 7fe5ae263640 10 monclient: shutdown 2024-03-29T10:52:57.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.826+0000 7fe5ae263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:52:57.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.826+0000 7fe5ae263640 1 -- 172.21.15.98:0/1294004181 >> v1:172.21.15.98:6789/0 conn(0x7fe5a80ae0e0 legacy=0x7fe5a806f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:52:57.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.828+0000 7fe5ae263640 1 -- 172.21.15.98:0/1294004181 shutdown_connections 2024-03-29T10:52:57.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.828+0000 7fe5ae263640 1 -- 172.21.15.98:0/1294004181 >> 172.21.15.98:0/1294004181 conn(0x7fe5a80b36e0 msgr2=0x7fe5a80aa250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:52:57.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.828+0000 7fe5ae263640 1 -- 172.21.15.98:0/1294004181 shutdown_connections 2024-03-29T10:52:57.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.829+0000 7fe5ae263640 1 -- 172.21.15.98:0/1294004181 wait complete. 2024-03-29T10:52:57.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:52:57.829+0000 7fe5ae263640 1 librados: shutdown 2024-03-29T10:52:57.831 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:04.637 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:04.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.823+0000 7f4fee7c4640 10 monclient: get_monmap_and_config 2024-03-29T10:53:04.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.823+0000 7f4fee7c4640 10 monclient: build_initial_monmap 2024-03-29T10:53:04.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.823+0000 7f4fee7c4640 10 monclient: monmap: 2024-03-29T10:53:04.806 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:04.806 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:04.806 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:53:04.824450+0000 2024-03-29T10:53:04.806 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:53:04.824450+0000 2024-03-29T10:53:04.806 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:04.806 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:04.806 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:04.807 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:53:04.807 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:53:04.807 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:04.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.825+0000 7f4fee7c4640 1 Processor -- start 2024-03-29T10:53:04.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.825+0000 7f4fee7c4640 1 -- start start 2024-03-29T10:53:04.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.825+0000 7f4fee7c4640 10 monclient: init 2024-03-29T10:53:04.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fee7c4640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:04.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fee7c4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:04.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fee7c4640 10 monclient(hunting): picked mon.noname-c con 0x7f4fe80b2520 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:04.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fee7c4640 10 monclient(hunting): picked mon.noname-a con 0x7f4fe80b2900 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:04.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fee7c4640 10 monclient(hunting): picked mon.noname-b con 0x7f4fe80a9c70 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:04.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fee7c4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe8142170 con 0x7f4fe80b2900 2024-03-29T10:53:04.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fe7fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4fe80b2520 0x7f4fe80b1b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:60822/0 (socket says 172.21.15.98:60822) 2024-03-29T10:53:04.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fecd61640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f4fe80a9c70 0x7f4fe814d7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:44116/0 (socket says 172.21.15.98:44116) 2024-03-29T10:53:04.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fe7fff640 1 -- 172.21.15.98:0/3437128543 learned_addr learned my addr 172.21.15.98:0/3437128543 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:04.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fee7c4640 1 -- 172.21.15.98:0/3437128543 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe80b2260 con 0x7f4fe80b2520 2024-03-29T10:53:04.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fee7c4640 1 -- 172.21.15.98:0/3437128543 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe80aaf60 con 0x7f4fe80a9c70 2024-03-29T10:53:04.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fee7c4640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:04.809 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fee7c4640 10 monclient(hunting): authenticate will time out at 1405.023690s 2024-03-29T10:53:04.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fe6ffd640 1 -- 172.21.15.98:0/3437128543 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194735440 0 0) 0x7f4fe80aaf60 con 0x7f4fe80a9c70 2024-03-29T10:53:04.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fe6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:04.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fe6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:04.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:04.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.826+0000 7f4fe6ffd640 10 monclient(hunting): my global_id is 4568 2024-03-29T10:53:04.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.827+0000 7f4fe6ffd640 1 -- 172.21.15.98:0/3437128543 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd8003520 con 0x7f4fe80a9c70 2024-03-29T10:53:04.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.827+0000 7f4fe6ffd640 1 -- 172.21.15.98:0/3437128543 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992292355 0 0) 0x7f4fe80b2260 con 0x7f4fe80b2520 2024-03-29T10:53:04.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.827+0000 7f4fe6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:04.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.827+0000 7f4fe6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:04.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.827+0000 7f4fe6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:04.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.827+0000 7f4fe6ffd640 10 monclient(hunting): my global_id is 4525 2024-03-29T10:53:04.810 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.827+0000 7f4fe6ffd640 1 -- 172.21.15.98:0/3437128543 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe80aaf60 con 0x7f4fe80b2520 2024-03-29T10:53:04.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.827+0000 7f4fe6ffd640 1 -- 172.21.15.98:0/3437128543 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3938899659 0 0) 0x7f4fd8003520 con 0x7f4fe80a9c70 2024-03-29T10:53:04.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.827+0000 7f4fe6ffd640 1 -- 172.21.15.98:0/3437128543 >> v1:172.21.15.189:6789/0 conn(0x7f4fe80b2520 legacy=0x7f4fe80b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:04.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.827+0000 7f4fe6ffd640 1 -- 172.21.15.98:0/3437128543 >> v1:172.21.15.98:6789/0 conn(0x7f4fe80b2900 legacy=0x7f4fe8142600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:04.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fe6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:04.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fe6ffd640 1 monclient: found mon.noname-b 2024-03-29T10:53:04.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fe6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:04.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fe6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:53:04.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fe6ffd640 1 -- 172.21.15.98:0/3437128543 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe8142d10 con 0x7f4fe80a9c70 2024-03-29T10:53:04.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fe6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:04.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fe6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:04.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:04.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fe6ffd640 1 -- 172.21.15.98:0/3437128543 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4fe0001a40 con 0x7f4fe80a9c70 2024-03-29T10:53:04.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:04.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fe6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:04.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fe6ffd640 10 monclient: dump: 2024-03-29T10:53:04.812 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:04.812 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:04.812 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:04.813 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:04.813 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:04.813 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:04.813 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:04.813 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:04.813 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:04.813 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:04.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fee7c4640 5 monclient: authenticate success, global_id 4568 2024-03-29T10:53:04.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.828+0000 7f4fee7c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:04.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.829+0000 7f4fe6ffd640 1 -- 172.21.15.98:0/3437128543 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fe0001ba0 con 0x7f4fe80a9c70 2024-03-29T10:53:04.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.829+0000 7f4fe6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:04.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.829+0000 7f4fe6ffd640 1 -- 172.21.15.98:0/3437128543 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4fe0001fb0 con 0x7f4fe80a9c70 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.829+0000 7f4fe6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.829+0000 7f4fe6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.829+0000 7f4fe6ffd640 10 monclient: dump: 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:04.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.829+0000 7f4fee7c4640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:04.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.829+0000 7f4fee7c4640 4 set_mon_vals no callback set 2024-03-29T10:53:04.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.830+0000 7f4fee7c4640 10 monclient: shutdown 2024-03-29T10:53:04.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.830+0000 7f4fee7c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:04.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.830+0000 7f4fee7c4640 1 -- 172.21.15.98:0/3437128543 >> v1:172.21.15.98:6790/0 conn(0x7f4fe80a9c70 legacy=0x7f4fe814d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:04.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.831+0000 7f4fee7c4640 1 -- 172.21.15.98:0/3437128543 shutdown_connections 2024-03-29T10:53:04.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.831+0000 7f4fee7c4640 1 -- 172.21.15.98:0/3437128543 >> 172.21.15.98:0/3437128543 conn(0x7f4fe806d180 msgr2=0x7f4fe8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:04.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.832+0000 7f4fee7c4640 1 -- 172.21.15.98:0/3437128543 shutdown_connections 2024-03-29T10:53:04.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.832+0000 7f4fee7c4640 1 -- 172.21.15.98:0/3437128543 wait complete. 2024-03-29T10:53:04.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.834+0000 7f4fee7c4640 10 monclient: build_initial_monmap 2024-03-29T10:53:04.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.834+0000 7f4fee7c4640 10 monclient: monmap: 2024-03-29T10:53:04.822 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:04.822 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:04.822 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:53:04.822 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:53:04.822 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:04.822 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.834+0000 7f4fee7c4640 1 librados: starting msgr at 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.834+0000 7f4fee7c4640 1 librados: starting objecter 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.834+0000 7f4fee7c4640 1 Processor -- start 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.835+0000 7f4fee7c4640 1 -- start start 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.835+0000 7f4fee7c4640 1 librados: setting wanted keys 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.835+0000 7f4fee7c4640 1 librados: calling monclient init 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.835+0000 7f4fee7c4640 10 monclient: init 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fee7c4640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fee7c4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fee7c4640 10 monclient(hunting): picked mon.noname-a con 0x7f4fe80a9c70 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:04.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fee7c4640 10 monclient(hunting): picked mon.noname-c con 0x7f4fe80b2520 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:04.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fee7c4640 10 monclient(hunting): picked mon.noname-b con 0x7f4fe813b130 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:04.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fee7c4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe81d70e0 con 0x7f4fe80a9c70 2024-03-29T10:53:04.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fee7c4640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe813ad80 con 0x7f4fe813b130 2024-03-29T10:53:04.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fee7c4640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe8139b10 con 0x7f4fe80b2520 2024-03-29T10:53:04.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fee7c4640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:04.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fee7c4640 10 monclient(hunting): authenticate will time out at 1405.033078s 2024-03-29T10:53:04.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fe77fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f4fe80b2520 0x7f4fe81d7570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:44730/0 (socket says 172.21.15.98:44730) 2024-03-29T10:53:04.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fe77fe640 1 -- 172.21.15.98:0/1247227608 learned_addr learned my addr 172.21.15.98:0/1247227608 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:04.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fe4ff9640 10 client.?.objecter ms_handle_connect 0x7f4fe80b2520 2024-03-29T10:53:04.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.836+0000 7f4fe4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:04.824 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.837+0000 7f4fe4ff9640 10 client.?.objecter ms_handle_connect 0x7f4fe813b130 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.837+0000 7f4fe4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.837+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162810546 0 0) 0x7f4fe8139b10 con 0x7f4fe80b2520 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.837+0000 7f4fe4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.837+0000 7f4fe4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.837+0000 7f4fe4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.837+0000 7f4fe4ff9640 10 monclient(hunting): my global_id is 4571 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.837+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fcc004eb0 con 0x7f4fe80b2520 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.837+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736062638 0 0) 0x7f4fe813ad80 con 0x7f4fe813b130 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 10 monclient(hunting): my global_id is 4528 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe8139b10 con 0x7f4fe813b130 2024-03-29T10:53:04.825 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3001435386 0 0) 0x7f4fcc004eb0 con 0x7f4fe80b2520 2024-03-29T10:53:04.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fe813ad80 con 0x7f4fe80b2520 2024-03-29T10:53:04.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4fd40017d0 con 0x7f4fe80b2520 2024-03-29T10:53:04.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:04.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:04.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 10 monclient(hunting): dump: 2024-03-29T10:53:04.826 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:04.827 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:04.827 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:04.827 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:04.827 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:04.827 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:04.827 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:04.827 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:04.827 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:04.827 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:04.827 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:04.827 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1456261670 0 0) 0x7f4fe8139b10 con 0x7f4fe813b130 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fcc004eb0 con 0x7f4fe813b130 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4fe0001f70 con 0x7f4fe813b130 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.838+0000 7f4fe4ff9640 10 monclient(hunting): dump: 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:04.828 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1392225424 0 0) 0x7f4fe813ad80 con 0x7f4fe80b2520 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 >> v1:172.21.15.189:6789/0 conn(0x7f4fe813b130 legacy=0x7f4fe813b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 >> v1:172.21.15.98:6789/0 conn(0x7f4fe80a9c70 legacy=0x7f4fe8139380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fe4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fe4ff9640 1 monclient: found mon.c 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fe4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fe4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe81d7c80 con 0x7f4fe80b2520 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fe4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fe4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fe4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fee7c4640 5 monclient: authenticate success, global_id 4571 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fee7c4640 10 monclient: _renew_subs 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fee7c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:04.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.839+0000 7f4fee7c4640 1 -- 172.21.15.98:0/1247227608 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe8145f50 con 0x7f4fe80b2520 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.840+0000 7f4fee7c4640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.840+0000 7f4fee7c4640 10 monclient: _renew_subs 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.840+0000 7f4fee7c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.840+0000 7f4fee7c4640 1 -- 172.21.15.98:0/1247227608 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe813bf10 con 0x7f4fe80b2520 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.840+0000 7f4fee7c4640 1 librados: init done 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.840+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fd4001930 con 0x7f4fe80b2520 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.840+0000 7f4fe4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.840+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4fd40049d0 con 0x7f4fe80b2520 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.840+0000 7f4fe4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.840+0000 7f4fe4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.840+0000 7f4fe4ff9640 10 monclient: dump: 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:04.830 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:04.831 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:04.831 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:04.831 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:04.831 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:04.831 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:04.831 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:04.831 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:04.831 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.841+0000 7f4fe67fc640 4 set_mon_vals no callback set 2024-03-29T10:53:04.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.841+0000 7f4fee7c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:04.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.842+0000 7f4fee7c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:04.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.842+0000 7f4fee7c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:04.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.842+0000 7f4fee7c4640 1 -- 172.21.15.98:0/1247227608 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc8005080 con 0x7f4fe80b2520 2024-03-29T10:53:04.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.843+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f4fd401a9a0 con 0x7f4fe80b2520 2024-03-29T10:53:04.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.843+0000 7f4fe7fff640 10 monclient: get_auth_request con 0x7f4fcc06de00 auth_method 0 2024-03-29T10:53:04.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.844+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f4fd40849d0 con 0x7f4fe80b2520 2024-03-29T10:53:04.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.844+0000 7f4fe4ff9640 10 client.4571.objecter ms_dispatch 0x7f4fe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:04.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.844+0000 7f4fe4ff9640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:04.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.844+0000 7f4fe4ff9640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:04.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.844+0000 7f4fe4ff9640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T10:53:04.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.844+0000 7f4fe4ff9640 10 client.4571.objecter ms_handle_connect 0x7f4fcc06de00 2024-03-29T10:53:04.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.851+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4fd40500e0 con 0x7f4fe80b2520 2024-03-29T10:53:04.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.851+0000 7f4fe4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:04.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:04.851+0000 7f4fe4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:05.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.216+0000 7f4fee7c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:05.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.216+0000 7f4fee7c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:05.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.216+0000 7f4fee7c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:05.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.216+0000 7f4fee7c4640 1 -- 172.21.15.98:0/1247227608 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fc8005330 con 0x7f4fe80b2520 2024-03-29T10:53:05.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.218+0000 7f4fe4ff9640 1 -- 172.21.15.98:0/1247227608 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f4fd40521e0 con 0x7f4fe80b2520 2024-03-29T10:53:05.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.218+0000 7f4fe4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:05.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.218+0000 7f4fe4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:05.200 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:53:05.200 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:05.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.222+0000 7f4fee7c4640 10 librados: watch_flush enter 2024-03-29T10:53:05.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.222+0000 7f4fee7c4640 10 librados: watch_flush exit 2024-03-29T10:53:05.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.222+0000 7f4fee7c4640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T10:53:05.204 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.222+0000 7f4fee7c4640 10 client.4571.objecter successfully canceled tick 2024-03-29T10:53:05.205 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.223+0000 7f4fee7c4640 1 -- 172.21.15.98:0/1247227608 >> v1:172.21.15.98:6816/1481799426 conn(0x7f4fcc06de00 legacy=0x7f4fcc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:05.205 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.223+0000 7f4fee7c4640 10 monclient: shutdown 2024-03-29T10:53:05.205 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.223+0000 7f4fee7c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:05.205 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.223+0000 7f4fee7c4640 1 -- 172.21.15.98:0/1247227608 >> v1:172.21.15.98:6790/0 conn(0x7f4fe80b2520 legacy=0x7f4fe81d7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:05.205 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.223+0000 7f4fee7c4640 1 -- 172.21.15.98:0/1247227608 shutdown_connections 2024-03-29T10:53:05.205 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.223+0000 7f4fee7c4640 1 -- 172.21.15.98:0/1247227608 >> 172.21.15.98:0/1247227608 conn(0x7f4fe806d180 msgr2=0x7f4fe8141490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:05.205 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.223+0000 7f4fee7c4640 1 -- 172.21.15.98:0/1247227608 shutdown_connections 2024-03-29T10:53:05.205 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.223+0000 7f4fee7c4640 1 -- 172.21.15.98:0/1247227608 wait complete. 2024-03-29T10:53:05.205 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:05.223+0000 7f4fee7c4640 1 librados: shutdown 2024-03-29T10:53:05.222 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:12.029 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:12.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.197+0000 7fb460ef6640 10 monclient: get_monmap_and_config 2024-03-29T10:53:12.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.197+0000 7fb460ef6640 10 monclient: build_initial_monmap 2024-03-29T10:53:12.179 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.197+0000 7fb460ef6640 10 monclient: monmap: 2024-03-29T10:53:12.179 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:12.179 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:12.179 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:53:12.198276+0000 2024-03-29T10:53:12.179 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:53:12.198276+0000 2024-03-29T10:53:12.179 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:12.179 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:12.179 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:12.179 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:53:12.179 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:53:12.179 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:12.180 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.198+0000 7fb460ef6640 1 Processor -- start 2024-03-29T10:53:12.180 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.198+0000 7fb460ef6640 1 -- start start 2024-03-29T10:53:12.180 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.198+0000 7fb460ef6640 10 monclient: init 2024-03-29T10:53:12.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.199+0000 7fb460ef6640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:12.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.199+0000 7fb460ef6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:12.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.199+0000 7fb460ef6640 10 monclient(hunting): picked mon.noname-c con 0x7fb45c0b4920 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:12.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.199+0000 7fb460ef6640 10 monclient(hunting): picked mon.noname-b con 0x7fb45c0b1790 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:12.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.199+0000 7fb460ef6640 10 monclient(hunting): picked mon.noname-a con 0x7fb45c146790 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:12.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.199+0000 7fb460ef6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c0b4070 con 0x7fb45c146790 2024-03-29T10:53:12.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.199+0000 7fb460ef6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c06fb80 con 0x7fb45c0b4920 2024-03-29T10:53:12.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.199+0000 7fb460ef6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c0b42c0 con 0x7fb45c0b1790 2024-03-29T10:53:12.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.199+0000 7fb460ef6640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:12.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.199+0000 7fb460ef6640 10 monclient(hunting): authenticate will time out at 1412.396659s 2024-03-29T10:53:12.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.199+0000 7fb459d9b640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fb45c0b1790 0x7fb45c0b1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:39954/0 (socket says 172.21.15.98:39954) 2024-03-29T10:53:12.181 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.199+0000 7fb459d9b640 1 -- 172.21.15.98:0/3098980871 learned_addr learned my addr 172.21.15.98:0/3098980871 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:12.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 1 -- 172.21.15.98:0/3098980871 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131660186 0 0) 0x7fb45c06fb80 con 0x7fb45c0b4920 2024-03-29T10:53:12.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 10 monclient(hunting): _negotiate 2024-03-29T10:53:12.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:12.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:12.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 10 monclient(hunting): my global_id is 4531 2024-03-29T10:53:12.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 1 -- 172.21.15.98:0/3098980871 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb444003520 con 0x7fb45c0b4920 2024-03-29T10:53:12.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 1 -- 172.21.15.98:0/3098980871 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4129407146 0 0) 0x7fb444003520 con 0x7fb45c0b4920 2024-03-29T10:53:12.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 1 -- 172.21.15.98:0/3098980871 >> v1:172.21.15.98:6790/0 conn(0x7fb45c0b1790 legacy=0x7fb45c0b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:12.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 1 -- 172.21.15.98:0/3098980871 >> v1:172.21.15.98:6789/0 conn(0x7fb45c146790 legacy=0x7fb45c146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:12.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 10 monclient: _finish_hunting 0 2024-03-29T10:53:12.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 1 monclient: found mon.noname-c 2024-03-29T10:53:12.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:12.185 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.189:6789/0 2024-03-29T10:53:12.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 1 -- 172.21.15.98:0/3098980871 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c0b22c0 con 0x7fb45c0b4920 2024-03-29T10:53:12.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.200+0000 7fb45959a640 10 monclient: _finish_auth 0 2024-03-29T10:53:12.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb45959a640 10 monclient: _check_auth_tickets 2024-03-29T10:53:12.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb45959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:12.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb45959a640 1 -- 172.21.15.98:0/3098980871 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb454001a10 con 0x7fb45c0b4920 2024-03-29T10:53:12.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb45959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:12.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb45959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:12.186 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb45959a640 10 monclient: dump: 2024-03-29T10:53:12.186 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:12.186 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:12.186 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:12.186 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:12.187 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:12.187 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:12.187 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:12.187 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:12.187 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:12.187 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:12.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb460ef6640 5 monclient: authenticate success, global_id 4531 2024-03-29T10:53:12.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb460ef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:12.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb45959a640 1 -- 172.21.15.98:0/3098980871 <== mon.1 v1:172.21.15.189:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb454001c30 con 0x7fb45c0b4920 2024-03-29T10:53:12.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb45959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:12.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb45959a640 1 -- 172.21.15.98:0/3098980871 <== mon.1 v1:172.21.15.189:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb454001fc0 con 0x7fb45c0b4920 2024-03-29T10:53:12.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb45959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:12.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb45959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb45959a640 10 monclient: dump: 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb460ef6640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.201+0000 7fb460ef6640 4 set_mon_vals no callback set 2024-03-29T10:53:12.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.202+0000 7fb460ef6640 10 monclient: shutdown 2024-03-29T10:53:12.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.202+0000 7fb460ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:12.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.202+0000 7fb460ef6640 1 -- 172.21.15.98:0/3098980871 >> v1:172.21.15.189:6789/0 conn(0x7fb45c0b4920 legacy=0x7fb45c06e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:12.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.202+0000 7fb460ef6640 1 -- 172.21.15.98:0/3098980871 shutdown_connections 2024-03-29T10:53:12.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.202+0000 7fb460ef6640 1 -- 172.21.15.98:0/3098980871 >> 172.21.15.98:0/3098980871 conn(0x7fb45c066a00 msgr2=0x7fb45c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:12.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.202+0000 7fb460ef6640 1 -- 172.21.15.98:0/3098980871 shutdown_connections 2024-03-29T10:53:12.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.202+0000 7fb460ef6640 1 -- 172.21.15.98:0/3098980871 wait complete. 2024-03-29T10:53:12.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.203+0000 7fb460ef6640 10 monclient: build_initial_monmap 2024-03-29T10:53:12.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.203+0000 7fb460ef6640 10 monclient: monmap: 2024-03-29T10:53:12.192 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:12.192 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:12.192 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:53:12.192 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:53:12.192 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:12.192 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:12.192 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:12.193 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:53:12.193 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:53:12.193 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:12.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.204+0000 7fb460ef6640 1 librados: starting msgr at 2024-03-29T10:53:12.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.204+0000 7fb460ef6640 1 librados: starting objecter 2024-03-29T10:53:12.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.204+0000 7fb460ef6640 1 Processor -- start 2024-03-29T10:53:12.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.204+0000 7fb460ef6640 1 -- start start 2024-03-29T10:53:12.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.204+0000 7fb460ef6640 1 librados: setting wanted keys 2024-03-29T10:53:12.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.204+0000 7fb460ef6640 1 librados: calling monclient init 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.204+0000 7fb460ef6640 10 monclient: init 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb460ef6640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb460ef6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb460ef6640 10 monclient(hunting): picked mon.noname-c con 0x7fb45c0b1790 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb460ef6640 10 monclient(hunting): picked mon.noname-b con 0x7fb45c0b4920 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb460ef6640 10 monclient(hunting): picked mon.noname-a con 0x7fb45c146790 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb460ef6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c137e70 con 0x7fb45c146790 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb460ef6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c1380e0 con 0x7fb45c0b4920 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb460ef6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c135a10 con 0x7fb45c0b1790 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb460ef6640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb460ef6640 10 monclient(hunting): authenticate will time out at 1412.402532s 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb45a59c640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fb45c0b1790 0x7fb45c136360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:44904/0 (socket says 172.21.15.98:44904) 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb45a59c640 1 -- 172.21.15.98:0/2044719010 learned_addr learned my addr 172.21.15.98:0/2044719010 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:12.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb44affd640 10 client.?.objecter ms_handle_connect 0x7fb45c0b1790 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb44affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb44affd640 10 client.?.objecter ms_handle_connect 0x7fb45c146790 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb44affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb44affd640 10 client.?.objecter ms_handle_connect 0x7fb45c0b4920 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.205+0000 7fb44affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.206+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341845950 0 0) 0x7fb45c135a10 con 0x7fb45c0b1790 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.206+0000 7fb44affd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.206+0000 7fb44affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.206+0000 7fb44affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.206+0000 7fb44affd640 10 monclient(hunting): my global_id is 4574 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.206+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb43c004eb0 con 0x7fb45c0b1790 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.206+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980739583 0 0) 0x7fb45c137e70 con 0x7fb45c146790 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.206+0000 7fb44affd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:12.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.206+0000 7fb44affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.206+0000 7fb44affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.206+0000 7fb44affd640 10 monclient(hunting): my global_id is 4515 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.206+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb45c135a10 con 0x7fb45c146790 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2221059553 0 0) 0x7fb45c135a10 con 0x7fb45c146790 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb45c137e70 con 0x7fb45c146790 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2680550565 0 0) 0x7fb43c004eb0 con 0x7fb45c0b1790 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb45c135a10 con 0x7fb45c0b1790 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb44c001810 con 0x7fb45c146790 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 10 monclient(hunting): dump: 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:12.196 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb454001f80 con 0x7fb45c0b1790 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 10 monclient(hunting): dump: 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:12.197 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:12.198 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:12.198 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:12.198 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:12.198 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:12.198 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:12.198 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:12.198 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:12.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1311327888 0 0) 0x7fb45c137e70 con 0x7fb45c146790 2024-03-29T10:53:12.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.207+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 >> v1:172.21.15.98:6790/0 conn(0x7fb45c0b1790 legacy=0x7fb45c136360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:12.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 >> v1:172.21.15.189:6789/0 conn(0x7fb45c0b4920 legacy=0x7fb45c136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:12.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:12.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 1 monclient: found mon.a 2024-03-29T10:53:12.198 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:12.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:12.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c135ba0 con 0x7fb45c146790 2024-03-29T10:53:12.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 10 monclient: _finish_auth 0 2024-03-29T10:53:12.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:12.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:12.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb460ef6640 5 monclient: authenticate success, global_id 4515 2024-03-29T10:53:12.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb460ef6640 10 monclient: _renew_subs 2024-03-29T10:53:12.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb460ef6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:12.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb460ef6640 1 -- 172.21.15.98:0/2044719010 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45c138500 con 0x7fb45c146790 2024-03-29T10:53:12.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb460ef6640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:12.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb460ef6640 10 monclient: _renew_subs 2024-03-29T10:53:12.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb460ef6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:12.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb460ef6640 1 -- 172.21.15.98:0/2044719010 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45c145940 con 0x7fb45c146790 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb460ef6640 1 librados: init done 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb44c001d20 con 0x7fb45c146790 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fb44c0037b0 con 0x7fb45c146790 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb44affd640 10 monclient: dump: 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:12.200 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:12.201 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:12.201 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:12.201 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:12.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.208+0000 7fb458d99640 4 set_mon_vals no callback set 2024-03-29T10:53:12.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.210+0000 7fb460ef6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:12.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.210+0000 7fb460ef6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:12.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.210+0000 7fb460ef6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:12.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.210+0000 7fb460ef6640 1 -- 172.21.15.98:0/2044719010 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb440005080 con 0x7fb45c146790 2024-03-29T10:53:12.201 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.210+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fb44c019780 con 0x7fb45c146790 2024-03-29T10:53:12.202 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.211+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fb44c0836c0 con 0x7fb45c146790 2024-03-29T10:53:12.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.211+0000 7fb44affd640 10 client.4515.objecter ms_dispatch 0x7fb45c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:12.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.211+0000 7fb44affd640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:12.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.211+0000 7fb44affd640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:12.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.211+0000 7fb45a59c640 10 monclient: get_auth_request con 0x7fb43c06de50 auth_method 0 2024-03-29T10:53:12.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.211+0000 7fb44affd640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-29T10:53:12.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.211+0000 7fb44affd640 10 client.4515.objecter ms_handle_connect 0x7fb43c06de50 2024-03-29T10:53:12.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.219+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb44c04f000 con 0x7fb45c146790 2024-03-29T10:53:12.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.219+0000 7fb44affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:12.203 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.219+0000 7fb44affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:12.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.542+0000 7fb460ef6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:12.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.542+0000 7fb460ef6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:12.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.542+0000 7fb460ef6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:12.525 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:53:12.525 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:12.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.542+0000 7fb460ef6640 1 -- 172.21.15.98:0/2044719010 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb440005330 con 0x7fb45c146790 2024-03-29T10:53:12.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.543+0000 7fb44affd640 1 -- 172.21.15.98:0/2044719010 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fb44c082d70 con 0x7fb45c146790 2024-03-29T10:53:12.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.543+0000 7fb44affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:12.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.543+0000 7fb44affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:12.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.546+0000 7fb460ef6640 10 librados: watch_flush enter 2024-03-29T10:53:12.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.546+0000 7fb460ef6640 10 librados: watch_flush exit 2024-03-29T10:53:12.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.546+0000 7fb460ef6640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-29T10:53:12.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.546+0000 7fb460ef6640 10 client.4515.objecter successfully canceled tick 2024-03-29T10:53:12.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.546+0000 7fb460ef6640 1 -- 172.21.15.98:0/2044719010 >> v1:172.21.15.98:6816/1481799426 conn(0x7fb43c06de50 legacy=0x7fb43c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:12.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.546+0000 7fb460ef6640 10 monclient: shutdown 2024-03-29T10:53:12.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.546+0000 7fb460ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:12.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.546+0000 7fb460ef6640 1 -- 172.21.15.98:0/2044719010 >> v1:172.21.15.98:6789/0 conn(0x7fb45c146790 legacy=0x7fb45c1e6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:12.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.546+0000 7fb460ef6640 1 -- 172.21.15.98:0/2044719010 shutdown_connections 2024-03-29T10:53:12.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.546+0000 7fb460ef6640 1 -- 172.21.15.98:0/2044719010 >> 172.21.15.98:0/2044719010 conn(0x7fb45c066a00 msgr2=0x7fb45c06ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:12.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.547+0000 7fb460ef6640 1 -- 172.21.15.98:0/2044719010 shutdown_connections 2024-03-29T10:53:12.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.547+0000 7fb460ef6640 1 -- 172.21.15.98:0/2044719010 wait complete. 2024-03-29T10:53:12.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:12.547+0000 7fb460ef6640 1 librados: shutdown 2024-03-29T10:53:12.542 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:19.349 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:19.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.524+0000 7f4520b96640 10 monclient: get_monmap_and_config 2024-03-29T10:53:19.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.524+0000 7f4520b96640 10 monclient: build_initial_monmap 2024-03-29T10:53:19.505 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.524+0000 7f4520b96640 10 monclient: monmap: 2024-03-29T10:53:19.506 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:19.506 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:19.506 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:53:19.524930+0000 2024-03-29T10:53:19.506 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:53:19.524930+0000 2024-03-29T10:53:19.506 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:19.506 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:19.506 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:19.506 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:53:19.506 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:53:19.506 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:19.508 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.525+0000 7f4520b96640 1 Processor -- start 2024-03-29T10:53:19.508 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.525+0000 7f4520b96640 1 -- start start 2024-03-29T10:53:19.508 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.525+0000 7f4520b96640 10 monclient: init 2024-03-29T10:53:19.508 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.526+0000 7f4520b96640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:19.508 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.526+0000 7f4520b96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:53:19.508 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.526+0000 7f4520b96640 10 monclient(hunting): picked mon.noname-c con 0x7f451c1422b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:19.508 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.526+0000 7f4520b96640 10 monclient(hunting): picked mon.noname-a con 0x7f451c142690 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:19.508 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.526+0000 7f4520b96640 10 monclient(hunting): picked mon.noname-b con 0x7f451c0b17d0 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:19.509 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.526+0000 7f4520b96640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c142c80 con 0x7f451c142690 2024-03-29T10:53:19.509 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.526+0000 7f4520b96640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0aae40 con 0x7f451c1422b0 2024-03-29T10:53:19.509 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.526+0000 7f4520b96640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c13fbd0 con 0x7f451c0b17d0 2024-03-29T10:53:19.509 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.526+0000 7f4520b96640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:19.509 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.526+0000 7f4520b96640 10 monclient(hunting): authenticate will time out at 1419.723476s 2024-03-29T10:53:19.509 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.526+0000 7f4519d9b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f451c142690 0x7f451c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39240/0 (socket says 172.21.15.98:39240) 2024-03-29T10:53:19.509 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.526+0000 7f4519d9b640 1 -- 172.21.15.98:0/2120413796 learned_addr learned my addr 172.21.15.98:0/2120413796 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:19.511 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.527+0000 7f451959a640 1 -- 172.21.15.98:0/2120413796 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272894506 0 0) 0x7f451c142c80 con 0x7f451c142690 2024-03-29T10:53:19.511 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.527+0000 7f451959a640 10 monclient(hunting): _negotiate 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.527+0000 7f451959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.527+0000 7f451959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.527+0000 7f451959a640 10 monclient(hunting): my global_id is 4518 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.527+0000 7f451959a640 1 -- 172.21.15.98:0/2120413796 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4508003520 con 0x7f451c142690 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.527+0000 7f451959a640 1 -- 172.21.15.98:0/2120413796 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783517766 0 0) 0x7f451c0aae40 con 0x7f451c1422b0 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.527+0000 7f451959a640 10 monclient(hunting): _negotiate 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.527+0000 7f451959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.527+0000 7f451959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.527+0000 7f451959a640 10 monclient(hunting): my global_id is 4537 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 1 -- 172.21.15.98:0/2120413796 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f451c142c80 con 0x7f451c1422b0 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 1 -- 172.21.15.98:0/2120413796 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138855594 0 0) 0x7f451c13fbd0 con 0x7f451c0b17d0 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient(hunting): _negotiate 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:19.512 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:19.513 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient(hunting): my global_id is 4577 2024-03-29T10:53:19.513 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 1 -- 172.21.15.98:0/2120413796 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f451c0aae40 con 0x7f451c0b17d0 2024-03-29T10:53:19.513 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 1 -- 172.21.15.98:0/2120413796 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 215019996 0 0) 0x7f4508003520 con 0x7f451c142690 2024-03-29T10:53:19.513 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 1 -- 172.21.15.98:0/2120413796 >> v1:172.21.15.98:6790/0 conn(0x7f451c0b17d0 legacy=0x7f451c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:19.513 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 1 -- 172.21.15.98:0/2120413796 >> v1:172.21.15.189:6789/0 conn(0x7f451c1422b0 legacy=0x7f451c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:19.513 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient: _finish_hunting 0 2024-03-29T10:53:19.513 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 1 monclient: found mon.noname-a 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 1 -- 172.21.15.98:0/2120413796 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c151940 con 0x7f451c142690 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient: _finish_auth 0 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient: _check_auth_tickets 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 1 -- 172.21.15.98:0/2120413796 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4514001a10 con 0x7f451c142690 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f4520b96640 5 monclient: authenticate success, global_id 4518 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f4520b96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient: dump: 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:19.514 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f4520b96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 1 -- 172.21.15.98:0/2120413796 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4514001c30 con 0x7f451c142690 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 1 -- 172.21.15.98:0/2120413796 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f4514001f40 con 0x7f451c142690 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f4520b96640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:19.515 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f4520b96640 4 set_mon_vals no callback set 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.528+0000 7f451959a640 10 monclient: dump: 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.529+0000 7f4520b96640 10 monclient: shutdown 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.529+0000 7f4520b96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:19.516 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.529+0000 7f4520b96640 1 -- 172.21.15.98:0/2120413796 >> v1:172.21.15.98:6789/0 conn(0x7f451c142690 legacy=0x7f451c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:19.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.529+0000 7f4520b96640 1 -- 172.21.15.98:0/2120413796 shutdown_connections 2024-03-29T10:53:19.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.529+0000 7f4520b96640 1 -- 172.21.15.98:0/2120413796 >> 172.21.15.98:0/2120413796 conn(0x7f451c066a00 msgr2=0x7f451c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:19.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.529+0000 7f4520b96640 1 -- 172.21.15.98:0/2120413796 shutdown_connections 2024-03-29T10:53:19.517 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.529+0000 7f4520b96640 1 -- 172.21.15.98:0/2120413796 wait complete. 2024-03-29T10:53:19.519 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.531+0000 7f4520b96640 10 monclient: build_initial_monmap 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.531+0000 7f4520b96640 10 monclient: monmap: 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.531+0000 7f4520b96640 1 librados: starting msgr at 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.531+0000 7f4520b96640 1 librados: starting objecter 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.531+0000 7f4520b96640 1 Processor -- start 2024-03-29T10:53:19.520 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.531+0000 7f4520b96640 1 -- start start 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.531+0000 7f4520b96640 1 librados: setting wanted keys 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.531+0000 7f4520b96640 1 librados: calling monclient init 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.531+0000 7f4520b96640 10 monclient: init 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.532+0000 7f4520b96640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.532+0000 7f4520b96640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.532+0000 7f4520b96640 10 monclient(hunting): picked mon.noname-a con 0x7f451c0b17d0 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.532+0000 7f4520b96640 10 monclient(hunting): picked mon.noname-b con 0x7f451c1422b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.532+0000 7f4520b96640 10 monclient(hunting): picked mon.noname-c con 0x7f451c142690 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.532+0000 7f4520b96640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0b79a0 con 0x7f451c0b17d0 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.532+0000 7f4520b96640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0b6be0 con 0x7f451c1422b0 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.532+0000 7f4520b96640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0b6ec0 con 0x7f451c142690 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.532+0000 7f4520b96640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.532+0000 7f4520b96640 10 monclient(hunting): authenticate will time out at 1419.729619s 2024-03-29T10:53:19.521 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.532+0000 7f451a59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f451c0b17d0 0x7f451c1dee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39356/0 (socket says 172.21.15.98:39356) 2024-03-29T10:53:19.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.532+0000 7f451a59c640 1 -- 172.21.15.98:0/3017496590 learned_addr learned my addr 172.21.15.98:0/3017496590 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:19.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.533+0000 7f4512ffd640 10 client.?.objecter ms_handle_connect 0x7f451c0b17d0 2024-03-29T10:53:19.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.533+0000 7f4512ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:19.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.533+0000 7f4512ffd640 10 client.?.objecter ms_handle_connect 0x7f451c142690 2024-03-29T10:53:19.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.533+0000 7f4512ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:19.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.533+0000 7f4512ffd640 10 client.?.objecter ms_handle_connect 0x7f451c1422b0 2024-03-29T10:53:19.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.533+0000 7f4512ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:19.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.533+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365311027 0 0) 0x7f451c0b6ec0 con 0x7f451c142690 2024-03-29T10:53:19.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.533+0000 7f4512ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:19.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.533+0000 7f4512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:19.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.533+0000 7f4512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:19.522 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.533+0000 7f4512ffd640 10 monclient(hunting): my global_id is 4580 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.533+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44fc004eb0 con 0x7f451c142690 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134729766 0 0) 0x7f451c0b6be0 con 0x7f451c1422b0 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 10 monclient(hunting): my global_id is 4540 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f451c0b6ec0 con 0x7f451c1422b0 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903722056 0 0) 0x7f451c0b79a0 con 0x7f451c0b17d0 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 10 monclient(hunting): my global_id is 4521 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f451c0b6be0 con 0x7f451c0b17d0 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 216364812 0 0) 0x7f451c0b6ec0 con 0x7f451c1422b0 2024-03-29T10:53:19.523 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.534+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44fc005670 con 0x7f451c1422b0 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f45140018d0 con 0x7f451c1422b0 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 10 monclient(hunting): dump: 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:19.524 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:19.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:19.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2289711014 0 0) 0x7f44fc004eb0 con 0x7f451c142690 2024-03-29T10:53:19.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44fc005d20 con 0x7f451c142690 2024-03-29T10:53:19.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f450c001870 con 0x7f451c142690 2024-03-29T10:53:19.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:19.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:19.525 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 10 monclient(hunting): dump: 2024-03-29T10:53:19.525 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:19.525 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:19.525 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:19.525 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:19.525 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:19.526 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:19.526 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:19.526 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:19.526 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:19.526 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:19.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2705016551 0 0) 0x7f44fc005d20 con 0x7f451c142690 2024-03-29T10:53:19.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 >> v1:172.21.15.189:6789/0 conn(0x7f451c1422b0 legacy=0x7f451c14db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:19.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 >> v1:172.21.15.98:6789/0 conn(0x7f451c0b17d0 legacy=0x7f451c1dee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:19.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:19.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 1 monclient: found mon.c 2024-03-29T10:53:19.526 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c0b71a0 con 0x7f451c142690 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4512ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4520b96640 5 monclient: authenticate success, global_id 4580 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4520b96640 10 monclient: _renew_subs 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4520b96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4520b96640 1 -- 172.21.15.98:0/3017496590 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451c150d10 con 0x7f451c142690 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4520b96640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4520b96640 10 monclient: _renew_subs 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4520b96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4520b96640 1 -- 172.21.15.98:0/3017496590 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f451c141200 con 0x7f451c142690 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.535+0000 7f4520b96640 1 librados: init done 2024-03-29T10:53:19.527 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.536+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f450c001d00 con 0x7f451c142690 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.536+0000 7f4512ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.536+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f450c003830 con 0x7f451c142690 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.536+0000 7f4512ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.536+0000 7f4512ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.536+0000 7f4512ffd640 10 monclient: dump: 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:19.528 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.536+0000 7f4518d99640 4 set_mon_vals no callback set 2024-03-29T10:53:19.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.537+0000 7f4520b96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:19.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.537+0000 7f4520b96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:19.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.537+0000 7f4520b96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:19.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.537+0000 7f4520b96640 1 -- 172.21.15.98:0/3017496590 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4500005080 con 0x7f451c142690 2024-03-29T10:53:19.529 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.538+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f450c019800 con 0x7f451c142690 2024-03-29T10:53:19.530 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.539+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f450c083720 con 0x7f451c142690 2024-03-29T10:53:19.530 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.539+0000 7f4512ffd640 10 client.4580.objecter ms_dispatch 0x7f451c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:19.530 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.539+0000 7f4512ffd640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:19.530 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.539+0000 7f4512ffd640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:19.530 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.539+0000 7f4512ffd640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T10:53:19.530 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.539+0000 7f451a59c640 10 monclient: get_auth_request con 0x7f44fc06e560 auth_method 0 2024-03-29T10:53:19.530 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.540+0000 7f4512ffd640 10 client.4580.objecter ms_handle_connect 0x7f44fc06e560 2024-03-29T10:53:19.531 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.549+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f450c04ee30 con 0x7f451c142690 2024-03-29T10:53:19.531 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.549+0000 7f4512ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:19.531 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.549+0000 7f4512ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:19.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.976+0000 7f4520b96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:19.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.976+0000 7f4520b96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:19.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.976+0000 7f4520b96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:19.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.976+0000 7f4520b96640 1 -- 172.21.15.98:0/3017496590 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4500005330 con 0x7f451c142690 2024-03-29T10:53:19.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.977+0000 7f4512ffd640 1 -- 172.21.15.98:0/3017496590 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f450c050f30 con 0x7f451c142690 2024-03-29T10:53:19.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.977+0000 7f4512ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:19.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.977+0000 7f4512ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:19.959 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:53:19.959 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:19.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.981+0000 7f4520b96640 10 librados: watch_flush enter 2024-03-29T10:53:19.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.981+0000 7f4520b96640 10 librados: watch_flush exit 2024-03-29T10:53:19.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.981+0000 7f4520b96640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T10:53:19.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.982+0000 7f4520b96640 10 client.4580.objecter successfully canceled tick 2024-03-29T10:53:19.963 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.982+0000 7f4520b96640 1 -- 172.21.15.98:0/3017496590 >> v1:172.21.15.98:6816/1481799426 conn(0x7f44fc06e560 legacy=0x7f44fc070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:19.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.982+0000 7f4520b96640 10 monclient: shutdown 2024-03-29T10:53:19.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.982+0000 7f4520b96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:19.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.982+0000 7f4520b96640 1 -- 172.21.15.98:0/3017496590 >> v1:172.21.15.98:6790/0 conn(0x7f451c142690 legacy=0x7f451c1e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:19.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.982+0000 7f4520b96640 1 -- 172.21.15.98:0/3017496590 shutdown_connections 2024-03-29T10:53:19.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.982+0000 7f4520b96640 1 -- 172.21.15.98:0/3017496590 >> 172.21.15.98:0/3017496590 conn(0x7f451c066a00 msgr2=0x7f451c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:19.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.982+0000 7f4520b96640 1 -- 172.21.15.98:0/3017496590 shutdown_connections 2024-03-29T10:53:19.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.983+0000 7f4520b96640 1 -- 172.21.15.98:0/3017496590 wait complete. 2024-03-29T10:53:19.964 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:19.983+0000 7f4520b96640 1 librados: shutdown 2024-03-29T10:53:19.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:26.788 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:26.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.961+0000 7f2e5e98d640 10 monclient: get_monmap_and_config 2024-03-29T10:53:26.943 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.961+0000 7f2e5e98d640 10 monclient: build_initial_monmap 2024-03-29T10:53:26.944 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.961+0000 7f2e5e98d640 10 monclient: monmap: 2024-03-29T10:53:26.944 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:26.944 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:26.944 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:53:26.962572+0000 2024-03-29T10:53:26.944 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:53:26.962572+0000 2024-03-29T10:53:26.944 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:26.944 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:26.944 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:26.945 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:53:26.945 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:53:26.945 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:26.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.962+0000 7f2e5e98d640 1 Processor -- start 2024-03-29T10:53:26.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.962+0000 7f2e5e98d640 1 -- start start 2024-03-29T10:53:26.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.962+0000 7f2e5e98d640 10 monclient: init 2024-03-29T10:53:26.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.963+0000 7f2e5e98d640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:26.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.963+0000 7f2e5e98d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:26.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.963+0000 7f2e5e98d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e580a9930 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:26.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.963+0000 7f2e5e98d640 10 monclient(hunting): picked mon.noname-b con 0x7f2e58149dc0 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:26.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.963+0000 7f2e5e98d640 10 monclient(hunting): picked mon.noname-c con 0x7f2e5814ff40 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:26.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.963+0000 7f2e5e98d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e580aae60 con 0x7f2e580a9930 2024-03-29T10:53:26.947 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.963+0000 7f2e5e98d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e580b4310 con 0x7f2e5814ff40 2024-03-29T10:53:26.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.963+0000 7f2e5e98d640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e580afdd0 con 0x7f2e58149dc0 2024-03-29T10:53:26.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.963+0000 7f2e5e98d640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:26.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.963+0000 7f2e5e98d640 10 monclient(hunting): authenticate will time out at 1427.160663s 2024-03-29T10:53:26.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.964+0000 7f2e57fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2e580a9930 0x7f2e58143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51658/0 (socket says 172.21.15.98:51658) 2024-03-29T10:53:26.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.964+0000 7f2e57fff640 1 -- 172.21.15.98:0/1729570136 learned_addr learned my addr 172.21.15.98:0/1729570136 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:26.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.964+0000 7f2e56ffd640 1 -- 172.21.15.98:0/1729570136 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831229551 0 0) 0x7f2e580aae60 con 0x7f2e580a9930 2024-03-29T10:53:26.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.964+0000 7f2e56ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:26.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.964+0000 7f2e56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:26.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.964+0000 7f2e56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:26.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.964+0000 7f2e56ffd640 10 monclient(hunting): my global_id is 4524 2024-03-29T10:53:26.948 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.964+0000 7f2e56ffd640 1 -- 172.21.15.98:0/1729570136 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e40003520 con 0x7f2e580a9930 2024-03-29T10:53:26.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.965+0000 7f2e56ffd640 1 -- 172.21.15.98:0/1729570136 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3361810764 0 0) 0x7f2e40003520 con 0x7f2e580a9930 2024-03-29T10:53:26.949 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.965+0000 7f2e56ffd640 1 -- 172.21.15.98:0/1729570136 >> v1:172.21.15.98:6790/0 conn(0x7f2e58149dc0 legacy=0x7f2e5814a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:26.955 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 1 -- 172.21.15.98:0/1729570136 >> v1:172.21.15.189:6789/0 conn(0x7f2e5814ff40 legacy=0x7f2e58152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:26.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:26.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 1 monclient: found mon.noname-a 2024-03-29T10:53:26.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:26.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:53:26.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 1 -- 172.21.15.98:0/1729570136 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5814a950 con 0x7f2e580a9930 2024-03-29T10:53:26.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:26.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:26.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:26.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 1 -- 172.21.15.98:0/1729570136 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2e48001690 con 0x7f2e580a9930 2024-03-29T10:53:26.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:26.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:26.956 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 10 monclient: dump: 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 1 -- 172.21.15.98:0/1729570136 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e48001920 con 0x7f2e580a9930 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 1 -- 172.21.15.98:0/1729570136 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2e48001cb0 con 0x7f2e580a9930 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:26.957 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e56ffd640 10 monclient: dump: 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e5e98d640 5 monclient: authenticate success, global_id 4524 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e5e98d640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.966+0000 7f2e5e98d640 4 set_mon_vals no callback set 2024-03-29T10:53:26.958 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.967+0000 7f2e5e98d640 10 monclient: shutdown 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.967+0000 7f2e5e98d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.967+0000 7f2e5e98d640 1 -- 172.21.15.98:0/1729570136 >> v1:172.21.15.98:6789/0 conn(0x7f2e580a9930 legacy=0x7f2e58143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.967+0000 7f2e5e98d640 1 -- 172.21.15.98:0/1729570136 shutdown_connections 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.967+0000 7f2e5e98d640 1 -- 172.21.15.98:0/1729570136 >> 172.21.15.98:0/1729570136 conn(0x7f2e580b35e0 msgr2=0x7f2e580b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.967+0000 7f2e5e98d640 1 -- 172.21.15.98:0/1729570136 shutdown_connections 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.967+0000 7f2e5e98d640 1 -- 172.21.15.98:0/1729570136 wait complete. 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.968+0000 7f2e5e98d640 10 monclient: build_initial_monmap 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.968+0000 7f2e5e98d640 10 monclient: monmap: 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:26.959 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.969+0000 7f2e5e98d640 1 librados: starting msgr at 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.969+0000 7f2e5e98d640 1 librados: starting objecter 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.969+0000 7f2e5e98d640 1 Processor -- start 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.970+0000 7f2e5e98d640 1 -- start start 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.970+0000 7f2e5e98d640 1 librados: setting wanted keys 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.970+0000 7f2e5e98d640 1 librados: calling monclient init 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.970+0000 7f2e5e98d640 10 monclient: init 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.971+0000 7f2e5e98d640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.971+0000 7f2e5e98d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.971+0000 7f2e5e98d640 10 monclient(hunting): picked mon.noname-c con 0x7f2e580a9930 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.971+0000 7f2e5e98d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e58149dc0 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:26.960 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.971+0000 7f2e5e98d640 10 monclient(hunting): picked mon.noname-b con 0x7f2e5814ff40 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:26.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.971+0000 7f2e5e98d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e581dec00 con 0x7f2e58149dc0 2024-03-29T10:53:26.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.971+0000 7f2e5e98d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5813ff00 con 0x7f2e5814ff40 2024-03-29T10:53:26.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.971+0000 7f2e5e98d640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e58140090 con 0x7f2e580a9930 2024-03-29T10:53:26.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.971+0000 7f2e5e98d640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:26.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.971+0000 7f2e5e98d640 10 monclient(hunting): authenticate will time out at 1427.168407s 2024-03-29T10:53:26.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.972+0000 7f2e577fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2e58149dc0 0x7f2e58149080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51734/0 (socket says 172.21.15.98:51734) 2024-03-29T10:53:26.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.972+0000 7f2e577fe640 1 -- 172.21.15.98:0/2824229228 learned_addr learned my addr 172.21.15.98:0/2824229228 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:26.961 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.974+0000 7f2e54ff9640 10 client.?.objecter ms_handle_connect 0x7f2e58149dc0 2024-03-29T10:53:26.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.975+0000 7f2e54ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:26.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.975+0000 7f2e54ff9640 10 client.?.objecter ms_handle_connect 0x7f2e5814ff40 2024-03-29T10:53:26.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.975+0000 7f2e54ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:26.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.975+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037280802 0 0) 0x7f2e5813ff00 con 0x7f2e5814ff40 2024-03-29T10:53:26.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.975+0000 7f2e54ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.975+0000 7f2e54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.975+0000 7f2e54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.975+0000 7f2e54ff9640 10 monclient(hunting): my global_id is 4543 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.975+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e44004eb0 con 0x7f2e5814ff40 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452380024 0 0) 0x7f2e581dec00 con 0x7f2e58149dc0 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient(hunting): my global_id is 4527 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e5813ff00 con 0x7f2e58149dc0 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 client.?.objecter ms_handle_connect 0x7f2e580a9930 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1947297337 0 0) 0x7f2e5813ff00 con 0x7f2e58149dc0 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e581dec00 con 0x7f2e58149dc0 2024-03-29T10:53:26.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2e4c001920 con 0x7f2e58149dc0 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient(hunting): dump: 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:26.970 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924171674 0 0) 0x7f2e58140090 con 0x7f2e580a9930 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient(hunting): my global_id is 4583 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e5813ff00 con 0x7f2e580a9930 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 80072655 0 0) 0x7f2e581dec00 con 0x7f2e58149dc0 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 >> v1:172.21.15.98:6790/0 conn(0x7f2e580a9930 legacy=0x7f2e581df120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 >> v1:172.21.15.189:6789/0 conn(0x7f2e5814ff40 legacy=0x7f2e5813ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 1 monclient: found mon.a 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:26.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e581df830 con 0x7f2e58149dc0 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient: _finish_auth 0 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e5e98d640 5 monclient: authenticate success, global_id 4527 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e5e98d640 10 monclient: _renew_subs 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e5e98d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.976+0000 7f2e5e98d640 1 -- 172.21.15.98:0/2824229228 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e58149790 con 0x7f2e58149dc0 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.977+0000 7f2e5e98d640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.977+0000 7f2e5e98d640 10 monclient: _renew_subs 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.977+0000 7f2e5e98d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.977+0000 7f2e5e98d640 1 -- 172.21.15.98:0/2824229228 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e58149ae0 con 0x7f2e58149dc0 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.977+0000 7f2e5e98d640 1 librados: init done 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.977+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e4c001a80 con 0x7f2e58149dc0 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.977+0000 7f2e54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:26.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.977+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2e4c0038a0 con 0x7f2e58149dc0 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.977+0000 7f2e54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.977+0000 7f2e54ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.977+0000 7f2e54ff9640 10 monclient: dump: 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:26.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.977+0000 7f2e55ffb640 4 set_mon_vals no callback set 2024-03-29T10:53:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.978+0000 7f2e5e98d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.978+0000 7f2e5e98d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.978+0000 7f2e5e98d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.978+0000 7f2e5e98d640 1 -- 172.21.15.98:0/2824229228 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e38005080 con 0x7f2e58149dc0 2024-03-29T10:53:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.979+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f2e4c003ac0 con 0x7f2e58149dc0 2024-03-29T10:53:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.979+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f2e4c084a00 con 0x7f2e58149dc0 2024-03-29T10:53:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.979+0000 7f2e54ff9640 10 client.4527.objecter ms_dispatch 0x7f2e58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.979+0000 7f2e54ff9640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.979+0000 7f2e54ff9640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.980+0000 7f2e54ff9640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-29T10:53:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.980+0000 7f2e57fff640 10 monclient: get_auth_request con 0x7f2e4406ded0 auth_method 0 2024-03-29T10:53:26.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.980+0000 7f2e54ff9640 10 client.4527.objecter ms_handle_connect 0x7f2e4406ded0 2024-03-29T10:53:26.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.988+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e4c050110 con 0x7f2e58149dc0 2024-03-29T10:53:26.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.988+0000 7f2e54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:26.975 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:26.988+0000 7f2e54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:27.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.323+0000 7f2e5e98d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:27.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.323+0000 7f2e5e98d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:27.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.323+0000 7f2e5e98d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:27.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.323+0000 7f2e5e98d640 1 -- 172.21.15.98:0/2824229228 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e38005330 con 0x7f2e58149dc0 2024-03-29T10:53:27.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.325+0000 7f2e54ff9640 1 -- 172.21.15.98:0/2824229228 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f2e4c052210 con 0x7f2e58149dc0 2024-03-29T10:53:27.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.325+0000 7f2e54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:27.306 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:53:27.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.325+0000 7f2e54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:27.307 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:27.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.328+0000 7f2e5e98d640 10 librados: watch_flush enter 2024-03-29T10:53:27.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.328+0000 7f2e5e98d640 10 librados: watch_flush exit 2024-03-29T10:53:27.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.328+0000 7f2e5e98d640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-29T10:53:27.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.328+0000 7f2e5e98d640 10 client.4527.objecter successfully canceled tick 2024-03-29T10:53:27.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.329+0000 7f2e5e98d640 1 -- 172.21.15.98:0/2824229228 >> v1:172.21.15.98:6816/1481799426 conn(0x7f2e4406ded0 legacy=0x7f2e44070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:27.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.329+0000 7f2e5e98d640 10 monclient: shutdown 2024-03-29T10:53:27.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.329+0000 7f2e5e98d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:27.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.329+0000 7f2e5e98d640 1 -- 172.21.15.98:0/2824229228 >> v1:172.21.15.98:6789/0 conn(0x7f2e58149dc0 legacy=0x7f2e58149080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:27.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.329+0000 7f2e5e98d640 1 -- 172.21.15.98:0/2824229228 shutdown_connections 2024-03-29T10:53:27.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.329+0000 7f2e5e98d640 1 -- 172.21.15.98:0/2824229228 >> 172.21.15.98:0/2824229228 conn(0x7f2e580b35e0 msgr2=0x7f2e580b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:27.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.329+0000 7f2e5e98d640 1 -- 172.21.15.98:0/2824229228 shutdown_connections 2024-03-29T10:53:27.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.329+0000 7f2e5e98d640 1 -- 172.21.15.98:0/2824229228 wait complete. 2024-03-29T10:53:27.311 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:27.329+0000 7f2e5e98d640 1 librados: shutdown 2024-03-29T10:53:27.325 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:34.131 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:34.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.294+0000 7f23b0433640 10 monclient: get_monmap_and_config 2024-03-29T10:53:34.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.294+0000 7f23b0433640 10 monclient: build_initial_monmap 2024-03-29T10:53:34.276 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.294+0000 7f23b0433640 10 monclient: monmap: 2024-03-29T10:53:34.276 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:34.276 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:34.276 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:53:34.295628+0000 2024-03-29T10:53:34.276 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:53:34.295628+0000 2024-03-29T10:53:34.276 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:34.276 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:34.276 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:34.276 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:53:34.276 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:53:34.277 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:34.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.295+0000 7f23b0433640 1 Processor -- start 2024-03-29T10:53:34.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.295+0000 7f23b0433640 1 -- start start 2024-03-29T10:53:34.277 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.295+0000 7f23b0433640 10 monclient: init 2024-03-29T10:53:34.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.296+0000 7f23b0433640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:34.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.297+0000 7f23b0433640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:34.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.297+0000 7f23b0433640 10 monclient(hunting): picked mon.noname-a con 0x7f23a81422b0 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:34.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.297+0000 7f23b0433640 10 monclient(hunting): picked mon.noname-b con 0x7f23a8142690 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:34.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.297+0000 7f23b0433640 10 monclient(hunting): picked mon.noname-c con 0x7f23a80b17d0 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:34.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.297+0000 7f23b0433640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a8142c80 con 0x7f23a81422b0 2024-03-29T10:53:34.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.297+0000 7f23b0433640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a80aae40 con 0x7f23a80b17d0 2024-03-29T10:53:34.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.297+0000 7f23b0433640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a813fbd0 con 0x7f23a8142690 2024-03-29T10:53:34.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.297+0000 7f23b0433640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:34.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.297+0000 7f23b0433640 10 monclient(hunting): authenticate will time out at 1434.494652s 2024-03-29T10:53:34.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.299+0000 7f23ae1cf640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f23a81422b0 0x7f23a813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50248/0 (socket says 172.21.15.98:50248) 2024-03-29T10:53:34.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.299+0000 7f23ae1cf640 1 -- 172.21.15.98:0/3934897106 learned_addr learned my addr 172.21.15.98:0/3934897106 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:34.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.299+0000 7f23ad1cd640 1 -- 172.21.15.98:0/3934897106 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463317300 0 0) 0x7f23a8142c80 con 0x7f23a81422b0 2024-03-29T10:53:34.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.299+0000 7f23ad1cd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:34.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.299+0000 7f23ad1cd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:34.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.299+0000 7f23ad1cd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:34.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.300+0000 7f23ad1cd640 10 monclient(hunting): my global_id is 4530 2024-03-29T10:53:34.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.300+0000 7f23ad1cd640 1 -- 172.21.15.98:0/3934897106 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2394003520 con 0x7f23a81422b0 2024-03-29T10:53:34.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.301+0000 7f23ad1cd640 1 -- 172.21.15.98:0/3934897106 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1763579303 0 0) 0x7f2394003520 con 0x7f23a81422b0 2024-03-29T10:53:34.282 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.301+0000 7f23ad1cd640 1 -- 172.21.15.98:0/3934897106 >> v1:172.21.15.98:6790/0 conn(0x7f23a8142690 legacy=0x7f23a80b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:34.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.301+0000 7f23ad1cd640 1 -- 172.21.15.98:0/3934897106 >> v1:172.21.15.189:6789/0 conn(0x7f23a80b17d0 legacy=0x7f23a80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:34.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.301+0000 7f23ad1cd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:34.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.301+0000 7f23ad1cd640 1 monclient: found mon.noname-a 2024-03-29T10:53:34.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.301+0000 7f23ad1cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:34.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.301+0000 7f23ad1cd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:53:34.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23ad1cd640 1 -- 172.21.15.98:0/3934897106 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a8151940 con 0x7f23a81422b0 2024-03-29T10:53:34.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23ad1cd640 10 monclient: _finish_auth 0 2024-03-29T10:53:34.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23ad1cd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:34.283 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23ad1cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23b0433640 5 monclient: authenticate success, global_id 4530 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23b0433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23ad1cd640 1 -- 172.21.15.98:0/3934897106 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2398001690 con 0x7f23a81422b0 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23ad1cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23ad1cd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23ad1cd640 10 monclient: dump: 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:34.284 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:34.285 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:34.285 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23b0433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:34.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23ad1cd640 1 -- 172.21.15.98:0/3934897106 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23980018b0 con 0x7f23a81422b0 2024-03-29T10:53:34.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23ad1cd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:34.286 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23b0433640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.302+0000 7f23b0433640 4 set_mon_vals no callback set 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.303+0000 7f23ad1cd640 1 -- 172.21.15.98:0/3934897106 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2398001c40 con 0x7f23a81422b0 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.303+0000 7f23ad1cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.303+0000 7f23ad1cd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.303+0000 7f23ad1cd640 10 monclient: dump: 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:34.287 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.303+0000 7f23b0433640 10 monclient: shutdown 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.303+0000 7f23b0433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.303+0000 7f23b0433640 1 -- 172.21.15.98:0/3934897106 >> v1:172.21.15.98:6789/0 conn(0x7f23a81422b0 legacy=0x7f23a813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.303+0000 7f23b0433640 1 -- 172.21.15.98:0/3934897106 shutdown_connections 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.303+0000 7f23b0433640 1 -- 172.21.15.98:0/3934897106 >> 172.21.15.98:0/3934897106 conn(0x7f23a8066a00 msgr2=0x7f23a8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.303+0000 7f23b0433640 1 -- 172.21.15.98:0/3934897106 shutdown_connections 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.303+0000 7f23b0433640 1 -- 172.21.15.98:0/3934897106 wait complete. 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.305+0000 7f23b0433640 10 monclient: build_initial_monmap 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.305+0000 7f23b0433640 10 monclient: monmap: 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:34.288 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:34.289 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:53:34.289 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:53:34.289 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:34.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.305+0000 7f23b0433640 1 librados: starting msgr at 2024-03-29T10:53:34.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.305+0000 7f23b0433640 1 librados: starting objecter 2024-03-29T10:53:34.289 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.306+0000 7f23b0433640 1 Processor -- start 2024-03-29T10:53:34.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.309+0000 7f23b0433640 1 -- start start 2024-03-29T10:53:34.290 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.309+0000 7f23b0433640 1 librados: setting wanted keys 2024-03-29T10:53:34.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.309+0000 7f23b0433640 1 librados: calling monclient init 2024-03-29T10:53:34.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.309+0000 7f23b0433640 10 monclient: init 2024-03-29T10:53:34.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.310+0000 7f23b0433640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:34.291 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.310+0000 7f23b0433640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:53:34.294 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.313+0000 7f23b0433640 10 monclient(hunting): picked mon.noname-c con 0x7f23a80b17d0 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:34.294 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.313+0000 7f23ae1cf640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f23a80b17d0 0x7f23a81dec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:49731/0 (socket says 172.21.15.98:49731) 2024-03-29T10:53:34.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.313+0000 7f23ae1cf640 1 -- 172.21.15.98:0/4156216101 learned_addr learned my addr 172.21.15.98:0/4156216101 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:34.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.313+0000 7f23b0433640 10 monclient(hunting): picked mon.noname-b con 0x7f23a81422b0 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:34.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.313+0000 7f23b0433640 10 monclient(hunting): picked mon.noname-a con 0x7f23a8142690 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:34.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.313+0000 7f23b0433640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a806de90 con 0x7f23a8142690 2024-03-29T10:53:34.295 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.313+0000 7f23b0433640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a806f340 con 0x7f23a81422b0 2024-03-29T10:53:34.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.314+0000 7f239effd640 10 client.?.objecter ms_handle_connect 0x7f23a80b17d0 2024-03-29T10:53:34.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.314+0000 7f239effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:34.296 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.314+0000 7f239effd640 10 client.?.objecter ms_handle_connect 0x7f23a8142690 2024-03-29T10:53:34.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.314+0000 7f239effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:34.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.314+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648622894 0 0) 0x7f23a806de90 con 0x7f23a8142690 2024-03-29T10:53:34.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.314+0000 7f23b0433640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a81df360 con 0x7f23a80b17d0 2024-03-29T10:53:34.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.314+0000 7f23b0433640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:34.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.314+0000 7f23b0433640 10 monclient(hunting): authenticate will time out at 1434.511331s 2024-03-29T10:53:34.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.314+0000 7f239effd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:34.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.314+0000 7f239effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:34.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.314+0000 7f239effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:34.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.314+0000 7f239effd640 10 monclient(hunting): my global_id is 4533 2024-03-29T10:53:34.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.314+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f238c004eb0 con 0x7f23a8142690 2024-03-29T10:53:34.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.317+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372947497 0 0) 0x7f23a81df360 con 0x7f23a80b17d0 2024-03-29T10:53:34.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.317+0000 7f239effd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:34.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.317+0000 7f239effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:34.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.317+0000 7f239effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:34.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.317+0000 7f239effd640 10 monclient(hunting): my global_id is 4586 2024-03-29T10:53:34.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.317+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a806de90 con 0x7f23a80b17d0 2024-03-29T10:53:34.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.317+0000 7f239effd640 10 client.?.objecter ms_handle_connect 0x7f23a81422b0 2024-03-29T10:53:34.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.317+0000 7f239effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:34.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.317+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 839373713 0 0) 0x7f238c004eb0 con 0x7f23a8142690 2024-03-29T10:53:34.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.317+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23a81df360 con 0x7f23a8142690 2024-03-29T10:53:34.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.317+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190225263 0 0) 0x7f23a806f340 con 0x7f23a81422b0 2024-03-29T10:53:34.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:34.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:34.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:34.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 10 monclient(hunting): my global_id is 4549 2024-03-29T10:53:34.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f238c004eb0 con 0x7f23a81422b0 2024-03-29T10:53:34.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f23a4001760 con 0x7f23a8142690 2024-03-29T10:53:34.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:34.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:34.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 10 monclient(hunting): dump: 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1998889674 0 0) 0x7f23a806de90 con 0x7f23a80b17d0 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23a806f340 con 0x7f23a80b17d0 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1757581497 0 0) 0x7f238c004eb0 con 0x7f23a81422b0 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23a806de90 con 0x7f23a81422b0 2024-03-29T10:53:34.304 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 165670635 0 0) 0x7f23a81df360 con 0x7f23a8142690 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.318+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 >> v1:172.21.15.98:6790/0 conn(0x7f23a80b17d0 legacy=0x7f23a81dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 >> v1:172.21.15.189:6789/0 conn(0x7f23a81422b0 legacy=0x7f23a812ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f239effd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f239effd640 1 monclient: found mon.a 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f239effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f239effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a8131b00 con 0x7f23a8142690 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f239effd640 10 monclient: _finish_auth 0 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f239effd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f239effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f23b0433640 5 monclient: authenticate success, global_id 4533 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f23b0433640 10 monclient: _renew_subs 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f23b0433640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:34.305 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f23b0433640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a8071dc0 con 0x7f23a8142690 2024-03-29T10:53:34.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f23b0433640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:34.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f23b0433640 10 monclient: _renew_subs 2024-03-29T10:53:34.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f23b0433640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:34.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f23b0433640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a812f330 con 0x7f23a8142690 2024-03-29T10:53:34.306 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.319+0000 7f23b0433640 1 librados: init done 2024-03-29T10:53:34.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.323+0000 7f23b0433640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:34.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.323+0000 7f23b0433640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:34.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.323+0000 7f23b0433640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:34.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.323+0000 7f23b0433640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a806e230 con 0x7f23a8142690 2024-03-29T10:53:34.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.323+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23a4001d50 con 0x7f23a8142690 2024-03-29T10:53:34.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.323+0000 7f239effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:34.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.323+0000 7f239ffff640 4 set_mon_vals no callback set 2024-03-29T10:53:34.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.323+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f23a4003790 con 0x7f23a8142690 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.323+0000 7f239effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.323+0000 7f239effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.323+0000 7f239effd640 10 monclient: dump: 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.323+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f23a40039b0 con 0x7f23a8142690 2024-03-29T10:53:34.309 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.324+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f23a4083720 con 0x7f23a8142690 2024-03-29T10:53:34.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.324+0000 7f239effd640 10 client.4533.objecter ms_dispatch 0x7f23a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:34.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.324+0000 7f239effd640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:34.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.324+0000 7f239effd640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:34.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.324+0000 7f239effd640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-29T10:53:34.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.324+0000 7f23ae1cf640 10 monclient: get_auth_request con 0x7f238c06e2e0 auth_method 0 2024-03-29T10:53:34.310 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.325+0000 7f239effd640 10 client.4533.objecter ms_handle_connect 0x7f238c06e2e0 2024-03-29T10:53:34.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.333+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23a404f080 con 0x7f23a8142690 2024-03-29T10:53:34.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.333+0000 7f239effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:34.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.333+0000 7f239effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:34.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.667+0000 7f23b0433640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:34.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.667+0000 7f23b0433640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:34.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.667+0000 7f23b0433640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:34.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.667+0000 7f23b0433640 1 -- 172.21.15.98:0/4156216101 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23a80a97d0 con 0x7f23a8142690 2024-03-29T10:53:34.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.669+0000 7f239effd640 1 -- 172.21.15.98:0/4156216101 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f23a4050fb0 con 0x7f23a8142690 2024-03-29T10:53:34.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.669+0000 7f239effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:34.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.669+0000 7f239effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:34.651 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:53:34.651 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:34.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.672+0000 7f23b0433640 10 librados: watch_flush enter 2024-03-29T10:53:34.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.673+0000 7f23b0433640 10 librados: watch_flush exit 2024-03-29T10:53:34.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.673+0000 7f23b0433640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-29T10:53:34.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.673+0000 7f23b0433640 10 client.4533.objecter successfully canceled tick 2024-03-29T10:53:34.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.673+0000 7f23b0433640 1 -- 172.21.15.98:0/4156216101 >> v1:172.21.15.98:6816/1481799426 conn(0x7f238c06e2e0 legacy=0x7f238c0707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:34.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.673+0000 7f23b0433640 10 monclient: shutdown 2024-03-29T10:53:34.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.673+0000 7f23b0433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:34.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.673+0000 7f23b0433640 1 -- 172.21.15.98:0/4156216101 >> v1:172.21.15.98:6789/0 conn(0x7f23a8142690 legacy=0x7f23a81e6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:34.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.673+0000 7f23b0433640 1 -- 172.21.15.98:0/4156216101 shutdown_connections 2024-03-29T10:53:34.655 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.673+0000 7f23b0433640 1 -- 172.21.15.98:0/4156216101 >> 172.21.15.98:0/4156216101 conn(0x7f23a8066a00 msgr2=0x7f23a813b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:34.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.673+0000 7f23b0433640 1 -- 172.21.15.98:0/4156216101 shutdown_connections 2024-03-29T10:53:34.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.673+0000 7f23b0433640 1 -- 172.21.15.98:0/4156216101 wait complete. 2024-03-29T10:53:34.656 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:34.674+0000 7f23b0433640 1 librados: shutdown 2024-03-29T10:53:34.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:41.479 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:41.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.643+0000 7fbd8ce24640 10 monclient: get_monmap_and_config 2024-03-29T10:53:41.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.643+0000 7fbd8ce24640 10 monclient: build_initial_monmap 2024-03-29T10:53:41.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.643+0000 7fbd8ce24640 10 monclient: monmap: 2024-03-29T10:53:41.626 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:41.626 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:41.626 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:53:41.644789+0000 2024-03-29T10:53:41.626 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:53:41.644789+0000 2024-03-29T10:53:41.627 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:41.627 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:41.627 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:41.627 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:53:41.627 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:53:41.627 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:41.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.645+0000 7fbd8ce24640 1 Processor -- start 2024-03-29T10:53:41.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.645+0000 7fbd8ce24640 1 -- start start 2024-03-29T10:53:41.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.645+0000 7fbd8ce24640 10 monclient: init 2024-03-29T10:53:41.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.646+0000 7fbd8ce24640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:41.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.646+0000 7fbd8ce24640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:53:41.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.646+0000 7fbd8ce24640 10 monclient(hunting): picked mon.noname-c con 0x7fbd80034850 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:41.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.646+0000 7fbd8ce24640 10 monclient(hunting): picked mon.noname-b con 0x7fbd8002a640 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:41.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.646+0000 7fbd8ce24640 10 monclient(hunting): picked mon.noname-a con 0x7fbd800327d0 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:41.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.646+0000 7fbd8ce24640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8002c820 con 0x7fbd800327d0 2024-03-29T10:53:41.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.646+0000 7fbd8ce24640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8002a120 con 0x7fbd80034850 2024-03-29T10:53:41.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.646+0000 7fbd8ce24640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd800c75b0 con 0x7fbd8002a640 2024-03-29T10:53:41.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.646+0000 7fbd8ce24640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:41.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.646+0000 7fbd8ce24640 10 monclient(hunting): authenticate will time out at 1441.843738s 2024-03-29T10:53:41.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.646+0000 7fbd86ffd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fbd8002a640 0x7fbd800320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:47628/0 (socket says 172.21.15.98:47628) 2024-03-29T10:53:41.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.646+0000 7fbd86ffd640 1 -- 172.21.15.98:0/494141440 learned_addr learned my addr 172.21.15.98:0/494141440 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:41.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.647+0000 7fbd867fc640 1 -- 172.21.15.98:0/494141440 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639737445 0 0) 0x7fbd800c75b0 con 0x7fbd8002a640 2024-03-29T10:53:41.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.647+0000 7fbd867fc640 10 monclient(hunting): _negotiate 2024-03-29T10:53:41.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.647+0000 7fbd867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:41.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.647+0000 7fbd867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:41.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.647+0000 7fbd867fc640 10 monclient(hunting): my global_id is 4589 2024-03-29T10:53:41.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.647+0000 7fbd867fc640 1 -- 172.21.15.98:0/494141440 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd78003500 con 0x7fbd8002a640 2024-03-29T10:53:41.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.647+0000 7fbd867fc640 1 -- 172.21.15.98:0/494141440 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 395740121 0 0) 0x7fbd78003500 con 0x7fbd8002a640 2024-03-29T10:53:41.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.647+0000 7fbd867fc640 1 -- 172.21.15.98:0/494141440 >> v1:172.21.15.189:6789/0 conn(0x7fbd80034850 legacy=0x7fbd800c6ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:53:41.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd867fc640 1 -- 172.21.15.98:0/494141440 >> v1:172.21.15.98:6789/0 conn(0x7fbd800327d0 legacy=0x7fbd800cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:41.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd867fc640 10 monclient: _finish_hunting 0 2024-03-29T10:53:41.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd867fc640 1 monclient: found mon.noname-b 2024-03-29T10:53:41.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:41.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd867fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:53:41.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd867fc640 1 -- 172.21.15.98:0/494141440 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd80033110 con 0x7fbd8002a640 2024-03-29T10:53:41.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd867fc640 10 monclient: _finish_auth 0 2024-03-29T10:53:41.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd867fc640 10 monclient: _check_auth_tickets 2024-03-29T10:53:41.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:41.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd867fc640 1 -- 172.21.15.98:0/494141440 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fbd74004cb0 con 0x7fbd8002a640 2024-03-29T10:53:41.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:41.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:41.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd867fc640 10 monclient: dump: 2024-03-29T10:53:41.636 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:41.636 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:41.636 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:41.636 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:41.636 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:41.636 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:41.636 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:41.636 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:41.636 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:41.636 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:41.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd8ce24640 5 monclient: authenticate success, global_id 4589 2024-03-29T10:53:41.636 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.654+0000 7fbd8ce24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:41.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.655+0000 7fbd867fc640 1 -- 172.21.15.98:0/494141440 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd74004e80 con 0x7fbd8002a640 2024-03-29T10:53:41.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.655+0000 7fbd867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:41.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.655+0000 7fbd8ce24640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:41.637 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.655+0000 7fbd8ce24640 4 set_mon_vals no callback set 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.655+0000 7fbd867fc640 1 -- 172.21.15.98:0/494141440 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fbd740013f0 con 0x7fbd8002a640 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.655+0000 7fbd867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.655+0000 7fbd867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.655+0000 7fbd867fc640 10 monclient: dump: 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:41.638 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:41.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.655+0000 7fbd8ce24640 10 monclient: shutdown 2024-03-29T10:53:41.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.655+0000 7fbd8ce24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:41.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.655+0000 7fbd8ce24640 1 -- 172.21.15.98:0/494141440 >> v1:172.21.15.98:6790/0 conn(0x7fbd8002a640 legacy=0x7fbd800320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:41.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.656+0000 7fbd8ce24640 1 -- 172.21.15.98:0/494141440 shutdown_connections 2024-03-29T10:53:41.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.656+0000 7fbd8ce24640 1 -- 172.21.15.98:0/494141440 >> 172.21.15.98:0/494141440 conn(0x7fbd800058f0 msgr2=0x7fbd80005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:41.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.656+0000 7fbd8ce24640 1 -- 172.21.15.98:0/494141440 shutdown_connections 2024-03-29T10:53:41.639 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.656+0000 7fbd8ce24640 1 -- 172.21.15.98:0/494141440 wait complete. 2024-03-29T10:53:41.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.657+0000 7fbd8ce24640 10 monclient: build_initial_monmap 2024-03-29T10:53:41.640 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.657+0000 7fbd8ce24640 10 monclient: monmap: 2024-03-29T10:53:41.640 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:41.640 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:41.640 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:53:41.640 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:53:41.640 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:41.640 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:41.640 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.657+0000 7fbd8ce24640 1 librados: starting msgr at 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.657+0000 7fbd8ce24640 1 librados: starting objecter 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.657+0000 7fbd8ce24640 1 Processor -- start 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.657+0000 7fbd8ce24640 1 -- start start 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.658+0000 7fbd8ce24640 1 librados: setting wanted keys 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.658+0000 7fbd8ce24640 1 librados: calling monclient init 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.658+0000 7fbd8ce24640 10 monclient: init 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.658+0000 7fbd8ce24640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.658+0000 7fbd8ce24640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.658+0000 7fbd8ce24640 10 monclient(hunting): picked mon.noname-a con 0x7fbd8002a640 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.658+0000 7fbd8ce24640 10 monclient(hunting): picked mon.noname-c con 0x7fbd800327d0 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.658+0000 7fbd8ce24640 10 monclient(hunting): picked mon.noname-b con 0x7fbd80034850 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:41.641 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.658+0000 7fbd8ce24640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd800310f0 con 0x7fbd8002a640 2024-03-29T10:53:41.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.658+0000 7fbd8ce24640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8002da60 con 0x7fbd80034850 2024-03-29T10:53:41.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.658+0000 7fbd8ce24640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd80165040 con 0x7fbd800327d0 2024-03-29T10:53:41.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.658+0000 7fbd8ce24640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:41.642 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.658+0000 7fbd8ce24640 10 monclient(hunting): authenticate will time out at 1441.855769s 2024-03-29T10:53:41.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.661+0000 7fbd86ffd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fbd800327d0 0x7fbd80164220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:51752/0 (socket says 172.21.15.98:51752) 2024-03-29T10:53:41.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.661+0000 7fbd86ffd640 1 -- 172.21.15.98:0/195742083 learned_addr learned my addr 172.21.15.98:0/195742083 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:41.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.661+0000 7fbd73fff640 10 client.?.objecter ms_handle_connect 0x7fbd800327d0 2024-03-29T10:53:41.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.661+0000 7fbd73fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:41.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.661+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458184295 0 0) 0x7fbd80165040 con 0x7fbd800327d0 2024-03-29T10:53:41.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.661+0000 7fbd73fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:41.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.661+0000 7fbd73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:41.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.661+0000 7fbd73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:41.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.661+0000 7fbd73fff640 10 monclient(hunting): my global_id is 4592 2024-03-29T10:53:41.644 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.661+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd6c004eb0 con 0x7fbd800327d0 2024-03-29T10:53:41.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 client.?.objecter ms_handle_connect 0x7fbd8002a640 2024-03-29T10:53:41.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:41.648 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025844763 0 0) 0x7fbd800310f0 con 0x7fbd8002a640 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 monclient(hunting): _negotiate 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 monclient(hunting): my global_id is 4539 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd80165040 con 0x7fbd8002a640 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2929152408 0 0) 0x7fbd6c004eb0 con 0x7fbd800327d0 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd800310f0 con 0x7fbd800327d0 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fbd74004b70 con 0x7fbd800327d0 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 monclient(hunting): dump: 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:41.649 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3315754127 0 0) 0x7fbd800310f0 con 0x7fbd800327d0 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 >> v1:172.21.15.189:6789/0 conn(0x7fbd80034850 legacy=0x7fbd80164930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 >> v1:172.21.15.98:6789/0 conn(0x7fbd8002a640 legacy=0x7fbd800c0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 monclient: _finish_hunting 0 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 1 monclient: found mon.c 2024-03-29T10:53:41.650 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8002dcd0 con 0x7fbd800327d0 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 monclient: _finish_auth 0 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 10 monclient: _check_auth_tickets 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.664+0000 7fbd73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.665+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd74004cd0 con 0x7fbd800327d0 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.665+0000 7fbd73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.665+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fbd74001be0 con 0x7fbd800327d0 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.665+0000 7fbd73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.665+0000 7fbd73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.665+0000 7fbd73fff640 10 monclient: dump: 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:41.651 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.665+0000 7fbd857fa640 4 set_mon_vals no callback set 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.666+0000 7fbd8ce24640 5 monclient: authenticate success, global_id 4592 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.666+0000 7fbd8ce24640 10 monclient: _renew_subs 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.666+0000 7fbd8ce24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.666+0000 7fbd8ce24640 1 -- 172.21.15.98:0/195742083 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd800c0130 con 0x7fbd800327d0 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.666+0000 7fbd8ce24640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.666+0000 7fbd8ce24640 10 monclient: _renew_subs 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.666+0000 7fbd8ce24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.666+0000 7fbd8ce24640 1 -- 172.21.15.98:0/195742083 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd800312e0 con 0x7fbd800327d0 2024-03-29T10:53:41.652 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.666+0000 7fbd8ce24640 1 librados: init done 2024-03-29T10:53:41.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.668+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fbd7401af70 con 0x7fbd800327d0 2024-03-29T10:53:41.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.669+0000 7fbd8ce24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:41.653 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.669+0000 7fbd8ce24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:41.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.669+0000 7fbd8ce24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:41.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.669+0000 7fbd8ce24640 1 -- 172.21.15.98:0/195742083 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd80004cd0 con 0x7fbd800327d0 2024-03-29T10:53:41.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.670+0000 7fbd877fe640 10 monclient: get_auth_request con 0x7fbd6c06ddc0 auth_method 0 2024-03-29T10:53:41.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.670+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fbd74084ba0 con 0x7fbd800327d0 2024-03-29T10:53:41.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.670+0000 7fbd73fff640 10 client.4592.objecter ms_dispatch 0x7fbd88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:41.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.670+0000 7fbd73fff640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:41.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.670+0000 7fbd73fff640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:41.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.670+0000 7fbd73fff640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T10:53:41.654 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.671+0000 7fbd73fff640 10 client.4592.objecter ms_handle_connect 0x7fbd6c06ddc0 2024-03-29T10:53:41.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.678+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd74050480 con 0x7fbd800327d0 2024-03-29T10:53:41.659 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.678+0000 7fbd73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:41.660 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:41.678+0000 7fbd73fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:41.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.002+0000 7fbd8ce24640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:41.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.002+0000 7fbd8ce24640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:41.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.002+0000 7fbd8ce24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:53:41.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.002+0000 7fbd8ce24640 1 -- 172.21.15.98:0/195742083 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd800bf850 con 0x7fbd800327d0 2024-03-29T10:53:41.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.003+0000 7fbd73fff640 1 -- 172.21.15.98:0/195742083 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fbd74001d40 con 0x7fbd800327d0 2024-03-29T10:53:41.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.003+0000 7fbd73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:41.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.003+0000 7fbd73fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:41.984 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:53:41.985 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:41.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.006+0000 7fbd8ce24640 10 librados: watch_flush enter 2024-03-29T10:53:41.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.006+0000 7fbd8ce24640 10 librados: watch_flush exit 2024-03-29T10:53:41.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.006+0000 7fbd8ce24640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T10:53:41.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.006+0000 7fbd8ce24640 10 client.4592.objecter successfully canceled tick 2024-03-29T10:53:41.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.007+0000 7fbd8ce24640 1 -- 172.21.15.98:0/195742083 >> v1:172.21.15.98:6816/1481799426 conn(0x7fbd6c06ddc0 legacy=0x7fbd6c070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:41.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.007+0000 7fbd8ce24640 10 monclient: shutdown 2024-03-29T10:53:41.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.007+0000 7fbd8ce24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:41.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.007+0000 7fbd8ce24640 1 -- 172.21.15.98:0/195742083 >> v1:172.21.15.98:6790/0 conn(0x7fbd800327d0 legacy=0x7fbd80164220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:41.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.010+0000 7fbd8ce24640 1 -- 172.21.15.98:0/195742083 shutdown_connections 2024-03-29T10:53:41.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.010+0000 7fbd8ce24640 1 -- 172.21.15.98:0/195742083 >> 172.21.15.98:0/195742083 conn(0x7fbd800058f0 msgr2=0x7fbd80005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:41.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.010+0000 7fbd8ce24640 1 -- 172.21.15.98:0/195742083 shutdown_connections 2024-03-29T10:53:41.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.010+0000 7fbd8ce24640 1 -- 172.21.15.98:0/195742083 wait complete. 2024-03-29T10:53:41.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:42.012+0000 7fbd8ce24640 1 librados: shutdown 2024-03-29T10:53:42.010 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:48.817 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:48.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.987+0000 7f04fd174640 10 monclient: get_monmap_and_config 2024-03-29T10:53:48.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.987+0000 7f04fd174640 10 monclient: build_initial_monmap 2024-03-29T10:53:48.969 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.987+0000 7f04fd174640 10 monclient: monmap: 2024-03-29T10:53:48.969 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:48.969 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:48.969 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:53:48.988199+0000 2024-03-29T10:53:48.969 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:53:48.988199+0000 2024-03-29T10:53:48.969 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:48.969 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:48.970 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:48.970 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:53:48.970 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:53:48.970 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:48.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.988+0000 7f04fd174640 1 Processor -- start 2024-03-29T10:53:48.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.988+0000 7f04fd174640 1 -- start start 2024-03-29T10:53:48.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.988+0000 7f04fd174640 10 monclient: init 2024-03-29T10:53:48.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.989+0000 7f04fd174640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:48.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.989+0000 7f04fd174640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:48.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.989+0000 7f04fd174640 10 monclient(hunting): picked mon.noname-a con 0x7f04f0006a70 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:48.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.989+0000 7f04fd174640 10 monclient(hunting): picked mon.noname-b con 0x7f04f002b100 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:48.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.989+0000 7f04fd174640 10 monclient(hunting): picked mon.noname-c con 0x7f04f0029c30 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:48.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.989+0000 7f04fd174640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f002a270 con 0x7f04f0006a70 2024-03-29T10:53:48.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.989+0000 7f04fd174640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f002c440 con 0x7f04f0029c30 2024-03-29T10:53:48.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.989+0000 7f04fd174640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f00c9440 con 0x7f04f002b100 2024-03-29T10:53:48.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.989+0000 7f04fd174640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:48.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.989+0000 7f04fd174640 10 monclient(hunting): authenticate will time out at 1449.186841s 2024-03-29T10:53:48.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.992+0000 7f04f77fe640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f04f002b100 0x7f04f00cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:39504/0 (socket says 172.21.15.98:39504) 2024-03-29T10:53:48.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.992+0000 7f04f77fe640 1 -- 172.21.15.98:0/970797482 learned_addr learned my addr 172.21.15.98:0/970797482 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:48.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.992+0000 7f04f6ffd640 1 -- 172.21.15.98:0/970797482 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411788429 0 0) 0x7f04f00c9440 con 0x7f04f002b100 2024-03-29T10:53:48.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.992+0000 7f04f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:48.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.992+0000 7f04f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:48.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.992+0000 7f04f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:48.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.992+0000 7f04f6ffd640 10 monclient(hunting): my global_id is 4595 2024-03-29T10:53:48.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.992+0000 7f04f6ffd640 1 -- 172.21.15.98:0/970797482 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e8003520 con 0x7f04f002b100 2024-03-29T10:53:48.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 1 -- 172.21.15.98:0/970797482 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337399179 0 0) 0x7f04f002a270 con 0x7f04f0006a70 2024-03-29T10:53:48.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:53:48.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:48.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:48.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 10 monclient(hunting): my global_id is 4542 2024-03-29T10:53:48.974 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 1 -- 172.21.15.98:0/970797482 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f00c9440 con 0x7f04f0006a70 2024-03-29T10:53:48.976 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 1 -- 172.21.15.98:0/970797482 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2658803920 0 0) 0x7f04e8003520 con 0x7f04f002b100 2024-03-29T10:53:48.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 1 -- 172.21.15.98:0/970797482 >> v1:172.21.15.189:6789/0 conn(0x7f04f0029c30 legacy=0x7f04f00d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:48.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 1 -- 172.21.15.98:0/970797482 >> v1:172.21.15.98:6789/0 conn(0x7f04f0006a70 legacy=0x7f04f00c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:48.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:53:48.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 1 monclient: found mon.noname-b 2024-03-29T10:53:48.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:48.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:53:48.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 1 -- 172.21.15.98:0/970797482 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f00c95d0 con 0x7f04f002b100 2024-03-29T10:53:48.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:53:48.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:53:48.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:48.977 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 1 -- 172.21.15.98:0/970797482 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f04e40015c0 con 0x7f04f002b100 2024-03-29T10:53:48.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:48.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:48.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 10 monclient: dump: 2024-03-29T10:53:48.978 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:48.978 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:48.978 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:48.978 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:48.978 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:48.978 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:48.978 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:48.979 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:48.979 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:48.979 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:48.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.993+0000 7f04f6ffd640 1 -- 172.21.15.98:0/970797482 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04e40017e0 con 0x7f04f002b100 2024-03-29T10:53:48.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.994+0000 7f04f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:48.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.994+0000 7f04f6ffd640 1 -- 172.21.15.98:0/970797482 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f04e4001b70 con 0x7f04f002b100 2024-03-29T10:53:48.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.994+0000 7f04fd174640 5 monclient: authenticate success, global_id 4595 2024-03-29T10:53:48.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.994+0000 7f04fd174640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:48.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.994+0000 7f04fd174640 4 set_mon_vals no callback set 2024-03-29T10:53:48.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.995+0000 7f04fd174640 10 monclient: shutdown 2024-03-29T10:53:48.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.995+0000 7f04fd174640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:48.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.995+0000 7f04fd174640 1 -- 172.21.15.98:0/970797482 >> v1:172.21.15.98:6790/0 conn(0x7f04f002b100 legacy=0x7f04f00cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:48.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.995+0000 7f04f6ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:53:48.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.995+0000 7f04fd174640 1 -- 172.21.15.98:0/970797482 shutdown_connections 2024-03-29T10:53:48.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.995+0000 7f04fd174640 1 -- 172.21.15.98:0/970797482 >> 172.21.15.98:0/970797482 conn(0x7f04f00058f0 msgr2=0x7f04f0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:48.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.995+0000 7f04fd174640 1 -- 172.21.15.98:0/970797482 shutdown_connections 2024-03-29T10:53:48.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.995+0000 7f04fd174640 1 -- 172.21.15.98:0/970797482 wait complete. 2024-03-29T10:53:48.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.996+0000 7f04fd174640 10 monclient: build_initial_monmap 2024-03-29T10:53:48.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.996+0000 7f04fd174640 10 monclient: monmap: 2024-03-29T10:53:48.983 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:48.983 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:48.984 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:53:48.984 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:53:48.984 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:48.984 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:48.984 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:48.984 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:53:48.984 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:53:48.984 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:48.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.997+0000 7f04fd174640 1 librados: starting msgr at 2024-03-29T10:53:48.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.997+0000 7f04fd174640 1 librados: starting objecter 2024-03-29T10:53:48.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.997+0000 7f04fd174640 1 Processor -- start 2024-03-29T10:53:48.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.998+0000 7f04fd174640 1 -- start start 2024-03-29T10:53:48.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.998+0000 7f04fd174640 1 librados: setting wanted keys 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.998+0000 7f04fd174640 1 librados: calling monclient init 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.998+0000 7f04fd174640 10 monclient: init 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.999+0000 7f04fd174640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.999+0000 7f04fd174640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.999+0000 7f04fd174640 10 monclient(hunting): picked mon.noname-c con 0x7f04f0006a70 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.999+0000 7f04fd174640 10 monclient(hunting): picked mon.noname-a con 0x7f04f0029c30 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.999+0000 7f04fd174640 10 monclient(hunting): picked mon.noname-b con 0x7f04f00ccd50 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.999+0000 7f04fd174640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f00cdd00 con 0x7f04f0029c30 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.999+0000 7f04fd174640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f00cdfe0 con 0x7f04f00ccd50 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.999+0000 7f04fd174640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f00ce400 con 0x7f04f0006a70 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.999+0000 7f04fd174640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:48.999+0000 7f04fd174640 10 monclient(hunting): authenticate will time out at 1449.196608s 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.000+0000 7f04f7fff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f04f0006a70 0x7f04f00cb910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:40422/0 (socket says 172.21.15.98:40422) 2024-03-29T10:53:48.985 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.000+0000 7f04f7fff640 1 -- 172.21.15.98:0/1945450542 learned_addr learned my addr 172.21.15.98:0/1945450542 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:48.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.004+0000 7f04f4ff9640 10 client.?.objecter ms_handle_connect 0x7f04f0006a70 2024-03-29T10:53:48.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.004+0000 7f04f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:48.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.004+0000 7f04f4ff9640 10 client.?.objecter ms_handle_connect 0x7f04f00ccd50 2024-03-29T10:53:48.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.004+0000 7f04f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:48.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 10 client.?.objecter ms_handle_connect 0x7f04f0029c30 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447119704 0 0) 0x7f04f00cdd00 con 0x7f04f0029c30 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 10 monclient(hunting): my global_id is 4545 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04dc004eb0 con 0x7f04f0029c30 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123480861 0 0) 0x7f04f00ce400 con 0x7f04f0006a70 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 10 monclient(hunting): my global_id is 4598 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.005+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f00cdd00 con 0x7f04f0006a70 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4276694756 0 0) 0x7f04dc004eb0 con 0x7f04f0029c30 2024-03-29T10:53:48.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04f00ce400 con 0x7f04f0029c30 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f04e4001af0 con 0x7f04f0029c30 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 10 monclient(hunting): dump: 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:48.993 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1468939888 0 0) 0x7f04f00ce400 con 0x7f04f0029c30 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 >> v1:172.21.15.98:6790/0 conn(0x7f04f0006a70 legacy=0x7f04f00cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 >> v1:172.21.15.189:6789/0 conn(0x7f04f00ccd50 legacy=0x7f04f016b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 1 monclient: found mon.a 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.006+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f00ce590 con 0x7f04f0029c30 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04f4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04f4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04fd174640 5 monclient: authenticate success, global_id 4545 2024-03-29T10:53:48.994 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04fd174640 10 monclient: _renew_subs 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04fd174640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04fd174640 1 -- 172.21.15.98:0/1945450542 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f0158360 con 0x7f04f0029c30 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04fd174640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04fd174640 10 monclient: _renew_subs 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04fd174640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04fd174640 1 -- 172.21.15.98:0/1945450542 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f0158700 con 0x7f04f0029c30 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04fd174640 1 librados: init done 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04e4004a90 con 0x7f04f0029c30 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.007+0000 7f04f67fc640 4 set_mon_vals no callback set 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.009+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f04e4004da0 con 0x7f04f0029c30 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.009+0000 7f04f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.009+0000 7f04f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.009+0000 7f04f4ff9640 10 monclient: dump: 2024-03-29T10:53:48.995 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.009+0000 7f04fd174640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.009+0000 7f04fd174640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.009+0000 7f04fd174640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.009+0000 7f04fd174640 1 -- 172.21.15.98:0/1945450542 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d8005080 con 0x7f04f0029c30 2024-03-29T10:53:48.996 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.009+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f04e401ad70 con 0x7f04f0029c30 2024-03-29T10:53:48.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.010+0000 7f04f7fff640 10 monclient: get_auth_request con 0x7f04dc06dd30 auth_method 0 2024-03-29T10:53:48.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.011+0000 7f04f4ff9640 10 client.4545.objecter ms_handle_connect 0x7f04dc06dd30 2024-03-29T10:53:48.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.011+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f04e40849b0 con 0x7f04f0029c30 2024-03-29T10:53:48.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.011+0000 7f04f4ff9640 10 client.4545.objecter ms_dispatch 0x7f04f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:48.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.011+0000 7f04f4ff9640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:48.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.011+0000 7f04f4ff9640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:48.998 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.011+0000 7f04f4ff9640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-29T10:53:48.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.018+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04e4050340 con 0x7f04f0029c30 2024-03-29T10:53:48.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.018+0000 7f04f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:48.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.018+0000 7f04f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:49.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.349+0000 7f04fd174640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:49.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.349+0000 7f04fd174640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:49.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.349+0000 7f04fd174640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:53:49.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.349+0000 7f04fd174640 1 -- 172.21.15.98:0/1945450542 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04d8005330 con 0x7f04f0029c30 2024-03-29T10:53:49.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.350+0000 7f04f4ff9640 1 -- 172.21.15.98:0/1945450542 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f04e4052270 con 0x7f04f0029c30 2024-03-29T10:53:49.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.350+0000 7f04f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:49.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.350+0000 7f04f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:49.333 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:53:49.333 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:49.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.354+0000 7f04fd174640 10 librados: watch_flush enter 2024-03-29T10:53:49.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.354+0000 7f04fd174640 10 librados: watch_flush exit 2024-03-29T10:53:49.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.354+0000 7f04fd174640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-29T10:53:49.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.354+0000 7f04fd174640 10 client.4545.objecter successfully canceled tick 2024-03-29T10:53:49.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.355+0000 7f04fd174640 1 -- 172.21.15.98:0/1945450542 >> v1:172.21.15.98:6816/1481799426 conn(0x7f04dc06dd30 legacy=0x7f04dc0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:49.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.355+0000 7f04fd174640 10 monclient: shutdown 2024-03-29T10:53:49.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.355+0000 7f04fd174640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:49.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.355+0000 7f04fd174640 1 -- 172.21.15.98:0/1945450542 >> v1:172.21.15.98:6789/0 conn(0x7f04f0029c30 legacy=0x7f04f00cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:49.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.355+0000 7f04fd174640 1 -- 172.21.15.98:0/1945450542 shutdown_connections 2024-03-29T10:53:49.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.355+0000 7f04fd174640 1 -- 172.21.15.98:0/1945450542 >> 172.21.15.98:0/1945450542 conn(0x7f04f00058f0 msgr2=0x7f04f0033b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:49.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.355+0000 7f04fd174640 1 -- 172.21.15.98:0/1945450542 shutdown_connections 2024-03-29T10:53:49.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.355+0000 7f04fd174640 1 -- 172.21.15.98:0/1945450542 wait complete. 2024-03-29T10:53:49.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:49.355+0000 7f04fd174640 1 librados: shutdown 2024-03-29T10:53:49.353 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:53:56.160 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:53:56.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.327+0000 7f30d50c8640 10 monclient: get_monmap_and_config 2024-03-29T10:53:56.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.327+0000 7f30d50c8640 10 monclient: build_initial_monmap 2024-03-29T10:53:56.308 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.327+0000 7f30d50c8640 10 monclient: monmap: 2024-03-29T10:53:56.309 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:56.309 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:56.309 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:53:56.328288+0000 2024-03-29T10:53:56.309 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:53:56.328288+0000 2024-03-29T10:53:56.309 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:56.309 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:56.309 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:56.309 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:53:56.309 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:53:56.309 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:56.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.328+0000 7f30d50c8640 1 Processor -- start 2024-03-29T10:53:56.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.328+0000 7f30d50c8640 1 -- start start 2024-03-29T10:53:56.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.328+0000 7f30d50c8640 10 monclient: init 2024-03-29T10:53:56.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.329+0000 7f30d50c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:56.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.329+0000 7f30d50c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:53:56.314 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.329+0000 7f30d50c8640 10 monclient(hunting): picked mon.noname-a con 0x7f30d01457f0 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.329+0000 7f30d50c8640 10 monclient(hunting): picked mon.noname-b con 0x7f30d013f650 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.329+0000 7f30d50c8640 10 monclient(hunting): picked mon.noname-c con 0x7f30d00a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.329+0000 7f30d50c8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d00aaef0 con 0x7f30d01457f0 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.329+0000 7f30d50c8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d0145bd0 con 0x7f30d00a9d70 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.329+0000 7f30d50c8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d0145eb0 con 0x7f30d013f650 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.329+0000 7f30d50c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.329+0000 7f30d50c8640 10 monclient(hunting): authenticate will time out at 1456.526651s 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.329+0000 7f30ced9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f30d01457f0 0x7f30d013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38954/0 (socket says 172.21.15.98:38954) 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.329+0000 7f30ced9d640 1 -- 172.21.15.98:0/1559959241 learned_addr learned my addr 172.21.15.98:0/1559959241 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.329+0000 7f30c7fff640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f30d013f650 0x7f30d00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:47976/0 (socket says 172.21.15.98:47976) 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 1 -- 172.21.15.98:0/1559959241 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648048518 0 0) 0x7f30d00aaef0 con 0x7f30d01457f0 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient(hunting): _negotiate 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:56.315 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient(hunting): my global_id is 4548 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 1 -- 172.21.15.98:0/1559959241 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30bc003520 con 0x7f30d01457f0 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 1 -- 172.21.15.98:0/1559959241 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183554823 0 0) 0x7f30d0145eb0 con 0x7f30d013f650 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient(hunting): _negotiate 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient(hunting): my global_id is 4601 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 1 -- 172.21.15.98:0/1559959241 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d00aaef0 con 0x7f30d013f650 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 1 -- 172.21.15.98:0/1559959241 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1650560128 0 0) 0x7f30d00aaef0 con 0x7f30d013f650 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 1 -- 172.21.15.98:0/1559959241 >> v1:172.21.15.189:6789/0 conn(0x7f30d00a9d70 legacy=0x7f30d014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 1 -- 172.21.15.98:0/1559959241 >> v1:172.21.15.98:6789/0 conn(0x7f30d01457f0 legacy=0x7f30d013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient: _finish_hunting 0 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 1 monclient: found mon.noname-b 2024-03-29T10:53:56.316 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 1 -- 172.21.15.98:0/1559959241 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d0146340 con 0x7f30d013f650 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient: _finish_auth 0 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient: _check_auth_tickets 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 1 -- 172.21.15.98:0/1559959241 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f30b8001670 con 0x7f30d013f650 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30ce59c640 10 monclient: dump: 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:56.317 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:56.318 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:56.318 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:56.318 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:56.318 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:56.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30d50c8640 5 monclient: authenticate success, global_id 4601 2024-03-29T10:53:56.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.330+0000 7f30d50c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:53:56.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.331+0000 7f30ce59c640 1 -- 172.21.15.98:0/1559959241 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30b80017d0 con 0x7f30d013f650 2024-03-29T10:53:56.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.331+0000 7f30ce59c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:56.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.331+0000 7f30ce59c640 1 -- 172.21.15.98:0/1559959241 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f30b8001be0 con 0x7f30d013f650 2024-03-29T10:53:56.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.331+0000 7f30ce59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:56.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.331+0000 7f30ce59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:53:56.318 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.331+0000 7f30ce59c640 10 monclient: dump: 2024-03-29T10:53:56.318 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.331+0000 7f30d50c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.331+0000 7f30d50c8640 4 set_mon_vals no callback set 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.332+0000 7f30d50c8640 10 monclient: shutdown 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.332+0000 7f30d50c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.332+0000 7f30d50c8640 1 -- 172.21.15.98:0/1559959241 >> v1:172.21.15.98:6790/0 conn(0x7f30d013f650 legacy=0x7f30d00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:56.319 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.332+0000 7f30d50c8640 1 -- 172.21.15.98:0/1559959241 shutdown_connections 2024-03-29T10:53:56.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.332+0000 7f30d50c8640 1 -- 172.21.15.98:0/1559959241 >> 172.21.15.98:0/1559959241 conn(0x7f30d0066a00 msgr2=0x7f30d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:56.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.332+0000 7f30d50c8640 1 -- 172.21.15.98:0/1559959241 shutdown_connections 2024-03-29T10:53:56.320 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.332+0000 7f30d50c8640 1 -- 172.21.15.98:0/1559959241 wait complete. 2024-03-29T10:53:56.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.333+0000 7f30d50c8640 10 monclient: build_initial_monmap 2024-03-29T10:53:56.326 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.333+0000 7f30d50c8640 10 monclient: monmap: 2024-03-29T10:53:56.326 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:53:56.326 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:53:56.326 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:53:56.326 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.334+0000 7f30d50c8640 1 librados: starting msgr at 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.334+0000 7f30d50c8640 1 librados: starting objecter 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.334+0000 7f30d50c8640 1 Processor -- start 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.334+0000 7f30d50c8640 1 -- start start 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.334+0000 7f30d50c8640 1 librados: setting wanted keys 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.334+0000 7f30d50c8640 1 librados: calling monclient init 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.334+0000 7f30d50c8640 10 monclient: init 2024-03-29T10:53:56.327 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30d50c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30d50c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30d50c8640 10 monclient(hunting): picked mon.noname-c con 0x7f30d00a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30d50c8640 10 monclient(hunting): picked mon.noname-a con 0x7f30d013f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30d50c8640 10 monclient(hunting): picked mon.noname-b con 0x7f30d01457f0 addr v1:172.21.15.189:6789/0 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30d50c8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d01d7510 con 0x7f30d013f650 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30d50c8640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d00acfe0 con 0x7f30d01457f0 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30d50c8640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d00ad2c0 con 0x7f30d00a9d70 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30d50c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30d50c8640 10 monclient(hunting): authenticate will time out at 1456.532267s 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30c7fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f30d013f650 0x7f30d00b0d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39482/0 (socket says 172.21.15.98:39482) 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30c7fff640 1 -- 172.21.15.98:0/801639657 learned_addr learned my addr 172.21.15.98:0/801639657 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30c77fe640 10 client.?.objecter ms_handle_connect 0x7f30d013f650 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:56.328 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30c77fe640 10 client.?.objecter ms_handle_connect 0x7f30d00a9d70 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.335+0000 7f30c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.336+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226923872 0 0) 0x7f30d01d7510 con 0x7f30d013f650 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.336+0000 7f30c77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.336+0000 7f30c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.336+0000 7f30c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.336+0000 7f30c77fe640 10 monclient(hunting): my global_id is 4551 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.336+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b0004eb0 con 0x7f30d013f650 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.336+0000 7f30c77fe640 10 client.?.objecter ms_handle_connect 0x7f30d01457f0 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.336+0000 7f30c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352953329 0 0) 0x7f30d00ad2c0 con 0x7f30d00a9d70 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:56.329 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): my global_id is 4604 2024-03-29T10:53:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d01d7510 con 0x7f30d00a9d70 2024-03-29T10:53:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901993845 0 0) 0x7f30d00acfe0 con 0x7f30d01457f0 2024-03-29T10:53:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:53:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:53:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:53:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): my global_id is 4558 2024-03-29T10:53:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d00ad2c0 con 0x7f30d01457f0 2024-03-29T10:53:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3903899032 0 0) 0x7f30d00ad2c0 con 0x7f30d01457f0 2024-03-29T10:53:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30d00acfe0 con 0x7f30d01457f0 2024-03-29T10:53:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 512671965 0 0) 0x7f30b0004eb0 con 0x7f30d013f650 2024-03-29T10:53:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30d00ad2c0 con 0x7f30d013f650 2024-03-29T10:53:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f30c8001b00 con 0x7f30d01457f0 2024-03-29T10:53:56.330 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): dump: 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f30b8001630 con 0x7f30d013f650 2024-03-29T10:53:56.331 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:56.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:53:56.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient(hunting): dump: 2024-03-29T10:53:56.332 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:56.332 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:56.332 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:56.332 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:56.332 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:56.332 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:56.332 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:56.332 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:56.332 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:56.332 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:56.332 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1221928836 0 0) 0x7f30d00acfe0 con 0x7f30d01457f0 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 >> v1:172.21.15.98:6790/0 conn(0x7f30d00a9d70 legacy=0x7f30d01d79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 >> v1:172.21.15.98:6789/0 conn(0x7f30d013f650 legacy=0x7f30d00b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 monclient: found mon.b 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d0140100 con 0x7f30d01457f0 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient: _finish_auth 0 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.337+0000 7f30c77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30d50c8640 5 monclient: authenticate success, global_id 4558 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30d50c8640 10 monclient: _renew_subs 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30d50c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:53:56.333 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30d50c8640 1 -- 172.21.15.98:0/801639657 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d01d80b0 con 0x7f30d01457f0 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30d50c8640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30d50c8640 10 monclient: _renew_subs 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30d50c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30d50c8640 1 -- 172.21.15.98:0/801639657 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d00ac1c0 con 0x7f30d01457f0 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30d50c8640 1 librados: init done 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30c80020f0 con 0x7f30d01457f0 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f30c8003b10 con 0x7f30d01457f0 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30c77fe640 10 monclient: dump: 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:53:56.334 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.338+0000 7f30cdd9b640 4 set_mon_vals no callback set 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.339+0000 7f30d50c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.340+0000 7f30d50c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.340+0000 7f30d50c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.340+0000 7f30d50c8640 1 -- 172.21.15.98:0/801639657 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d01d76c0 con 0x7f30d01457f0 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.341+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f30c8019ae0 con 0x7f30d01457f0 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.341+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f30c8083a60 con 0x7f30d01457f0 2024-03-29T10:53:56.335 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.341+0000 7f30c77fe640 10 client.4558.objecter ms_dispatch 0x7f30d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:53:56.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.341+0000 7f30c77fe640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:53:56.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.341+0000 7f30c77fe640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:53:56.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.341+0000 7f30ced9d640 10 monclient: get_auth_request con 0x7f30b006e050 auth_method 0 2024-03-29T10:53:56.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.341+0000 7f30c77fe640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-29T10:53:56.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.342+0000 7f30c77fe640 10 client.4558.objecter ms_handle_connect 0x7f30b006e050 2024-03-29T10:53:56.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.350+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30d01d76c0 con 0x7f30d01457f0 2024-03-29T10:53:56.336 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.350+0000 7f30c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:53:56.337 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.350+0000 7f30c77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:53:56.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.726+0000 7f30d50c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:56.707 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.726+0000 7f30d50c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:56.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.726+0000 7f30d50c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:53:56.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.726+0000 7f30d50c8640 1 -- 172.21.15.98:0/801639657 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30d01d76c0 con 0x7f30d01457f0 2024-03-29T10:53:56.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.727+0000 7f30c77fe640 1 -- 172.21.15.98:0/801639657 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f30d01d76c0 con 0x7f30d01457f0 2024-03-29T10:53:56.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.727+0000 7f30c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:53:56.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.727+0000 7f30c77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:53:56.709 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:53:56.709 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:53:56.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.731+0000 7f30d50c8640 10 librados: watch_flush enter 2024-03-29T10:53:56.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.731+0000 7f30d50c8640 10 librados: watch_flush exit 2024-03-29T10:53:56.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.731+0000 7f30d50c8640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-29T10:53:56.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.731+0000 7f30d50c8640 10 client.4558.objecter successfully canceled tick 2024-03-29T10:53:56.713 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.731+0000 7f30d50c8640 1 -- 172.21.15.98:0/801639657 >> v1:172.21.15.98:6816/1481799426 conn(0x7f30b006e050 legacy=0x7f30b0070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:56.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.731+0000 7f30d50c8640 10 monclient: shutdown 2024-03-29T10:53:56.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.731+0000 7f30d50c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:53:56.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.731+0000 7f30d50c8640 1 -- 172.21.15.98:0/801639657 >> v1:172.21.15.189:6789/0 conn(0x7f30d01457f0 legacy=0x7f30d01e2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:53:56.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.732+0000 7f30d50c8640 1 -- 172.21.15.98:0/801639657 shutdown_connections 2024-03-29T10:53:56.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.732+0000 7f30d50c8640 1 -- 172.21.15.98:0/801639657 >> 172.21.15.98:0/801639657 conn(0x7f30d0066a00 msgr2=0x7f30d00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:53:56.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.732+0000 7f30d50c8640 1 -- 172.21.15.98:0/801639657 shutdown_connections 2024-03-29T10:53:56.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.732+0000 7f30d50c8640 1 -- 172.21.15.98:0/801639657 wait complete. 2024-03-29T10:53:56.714 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:53:56.732+0000 7f30d50c8640 1 librados: shutdown 2024-03-29T10:53:56.728 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:03.535 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:03.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.704+0000 7fc96f59e640 10 monclient: get_monmap_and_config 2024-03-29T10:54:03.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.704+0000 7fc96f59e640 10 monclient: build_initial_monmap 2024-03-29T10:54:03.685 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.704+0000 7fc96f59e640 10 monclient: monmap: 2024-03-29T10:54:03.686 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:03.686 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:03.686 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:54:03.705149+0000 2024-03-29T10:54:03.686 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:54:03.705149+0000 2024-03-29T10:54:03.686 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:03.686 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:03.686 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:03.686 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:54:03.686 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:54:03.686 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:03.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.705+0000 7fc96f59e640 1 Processor -- start 2024-03-29T10:54:03.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.706+0000 7fc96f59e640 1 -- start start 2024-03-29T10:54:03.687 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.706+0000 7fc96f59e640 10 monclient: init 2024-03-29T10:54:03.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.707+0000 7fc96f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:03.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.707+0000 7fc96f59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:03.688 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.707+0000 7fc96f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9680389e0 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:03.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.708+0000 7fc96f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc968038dc0 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:03.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.708+0000 7fc96f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc968030660 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:03.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.708+0000 7fc96f59e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9680c54c0 con 0x7fc9680389e0 2024-03-29T10:54:03.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.708+0000 7fc96f59e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc968036430 con 0x7fc968038dc0 2024-03-29T10:54:03.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.708+0000 7fc96f59e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc968039840 con 0x7fc968030660 2024-03-29T10:54:03.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.708+0000 7fc96f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:03.689 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.708+0000 7fc96f59e640 10 monclient(hunting): authenticate will time out at 1463.905050s 2024-03-29T10:54:03.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.711+0000 7fc96ed9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fc968030660 0x7fc9680d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:40423/0 (socket says 172.21.15.98:40423) 2024-03-29T10:54:03.692 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.711+0000 7fc96ed9d640 1 -- 172.21.15.98:0/2258586317 learned_addr learned my addr 172.21.15.98:0/2258586317 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 1 -- 172.21.15.98:0/2258586317 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162274666 0 0) 0x7fc968039840 con 0x7fc968030660 2024-03-29T10:54:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:54:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient(hunting): my global_id is 4607 2024-03-29T10:54:03.693 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 1 -- 172.21.15.98:0/2258586317 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc958003520 con 0x7fc968030660 2024-03-29T10:54:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 1 -- 172.21.15.98:0/2258586317 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1947795939 0 0) 0x7fc958003520 con 0x7fc968030660 2024-03-29T10:54:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 1 -- 172.21.15.98:0/2258586317 >> v1:172.21.15.189:6789/0 conn(0x7fc968038dc0 legacy=0x7fc9680cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 1 -- 172.21.15.98:0/2258586317 >> v1:172.21.15.98:6789/0 conn(0x7fc9680389e0 legacy=0x7fc968033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:54:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 1 monclient: found mon.noname-b 2024-03-29T10:54:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:54:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 1 -- 172.21.15.98:0/2258586317 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc968035690 con 0x7fc968030660 2024-03-29T10:54:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient: _finish_auth 0 2024-03-29T10:54:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:54:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 1 -- 172.21.15.98:0/2258586317 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fc97004f730 con 0x7fc968030660 2024-03-29T10:54:03.694 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient: dump: 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 1 -- 172.21.15.98:0/2258586317 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc970075f10 con 0x7fc968030660 2024-03-29T10:54:03.695 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 1 -- 172.21.15.98:0/2258586317 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fc97004f730 con 0x7fc968030660 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.712+0000 7fc96d59a640 10 monclient: dump: 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:03.696 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:03.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.715+0000 7fc96f59e640 5 monclient: authenticate success, global_id 4607 2024-03-29T10:54:03.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.715+0000 7fc96f59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:03.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.715+0000 7fc96f59e640 4 set_mon_vals no callback set 2024-03-29T10:54:03.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.716+0000 7fc96f59e640 10 monclient: shutdown 2024-03-29T10:54:03.697 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.716+0000 7fc96f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:03.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.716+0000 7fc96f59e640 1 -- 172.21.15.98:0/2258586317 >> v1:172.21.15.98:6790/0 conn(0x7fc968030660 legacy=0x7fc9680d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:03.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.716+0000 7fc96f59e640 1 -- 172.21.15.98:0/2258586317 shutdown_connections 2024-03-29T10:54:03.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.716+0000 7fc96f59e640 1 -- 172.21.15.98:0/2258586317 >> 172.21.15.98:0/2258586317 conn(0x7fc968005c00 msgr2=0x7fc968006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:03.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.717+0000 7fc96f59e640 1 -- 172.21.15.98:0/2258586317 shutdown_connections 2024-03-29T10:54:03.698 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.718+0000 7fc96f59e640 1 -- 172.21.15.98:0/2258586317 wait complete. 2024-03-29T10:54:03.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.719+0000 7fc96f59e640 10 monclient: build_initial_monmap 2024-03-29T10:54:03.700 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.719+0000 7fc96f59e640 10 monclient: monmap: 2024-03-29T10:54:03.700 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:03.700 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:03.700 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:54:03.700 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:54:03.700 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:03.700 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:03.700 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:03.700 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:54:03.700 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:54:03.701 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.719+0000 7fc96f59e640 1 librados: starting msgr at 2024-03-29T10:54:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.719+0000 7fc96f59e640 1 librados: starting objecter 2024-03-29T10:54:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.720+0000 7fc96f59e640 1 Processor -- start 2024-03-29T10:54:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.720+0000 7fc96f59e640 1 -- start start 2024-03-29T10:54:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.720+0000 7fc96f59e640 1 librados: setting wanted keys 2024-03-29T10:54:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.720+0000 7fc96f59e640 1 librados: calling monclient init 2024-03-29T10:54:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.720+0000 7fc96f59e640 10 monclient: init 2024-03-29T10:54:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.721+0000 7fc96f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.721+0000 7fc96f59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:03.702 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.721+0000 7fc96f59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc968030660 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:03.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.724+0000 7fc96f59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9680389e0 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:03.705 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.724+0000 7fc96f59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc96802f340 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:03.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.724+0000 7fc96f59e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96815d730 con 0x7fc9680389e0 2024-03-29T10:54:03.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.724+0000 7fc96f59e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9680c8710 con 0x7fc96802f340 2024-03-29T10:54:03.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.724+0000 7fc96f59e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9680c88a0 con 0x7fc968030660 2024-03-29T10:54:03.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.724+0000 7fc96f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:03.706 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.724+0000 7fc96f59e640 10 monclient(hunting): authenticate will time out at 1463.921905s 2024-03-29T10:54:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.726+0000 7fc96dd9b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc9680389e0 0x7fc9680c22a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51563/0 (socket says 172.21.15.98:51563) 2024-03-29T10:54:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.726+0000 7fc96dd9b640 1 -- 172.21.15.98:0/919120038 learned_addr learned my addr 172.21.15.98:0/919120038 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.726+0000 7fc95effd640 10 client.?.objecter ms_handle_connect 0x7fc9680389e0 2024-03-29T10:54:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.726+0000 7fc95effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.726+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381067098 0 0) 0x7fc96815d730 con 0x7fc9680389e0 2024-03-29T10:54:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.726+0000 7fc95effd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.726+0000 7fc95effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.726+0000 7fc95effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.726+0000 7fc95effd640 10 monclient(hunting): my global_id is 4554 2024-03-29T10:54:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.726+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc950004eb0 con 0x7fc9680389e0 2024-03-29T10:54:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.727+0000 7fc95effd640 10 client.?.objecter ms_handle_connect 0x7fc968030660 2024-03-29T10:54:03.708 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.727+0000 7fc95effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:03.709 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.728+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2604237778 0 0) 0x7fc950004eb0 con 0x7fc9680389e0 2024-03-29T10:54:03.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.728+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc96815d730 con 0x7fc9680389e0 2024-03-29T10:54:03.710 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.729+0000 7fc95effd640 10 client.?.objecter ms_handle_connect 0x7fc96802f340 2024-03-29T10:54:03.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:03.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214173710 0 0) 0x7fc9680c88a0 con 0x7fc968030660 2024-03-29T10:54:03.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:03.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:03.715 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:03.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 10 monclient(hunting): my global_id is 4610 2024-03-29T10:54:03.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc950004eb0 con 0x7fc968030660 2024-03-29T10:54:03.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fc964001960 con 0x7fc9680389e0 2024-03-29T10:54:03.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:03.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:03.716 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 10 monclient(hunting): dump: 2024-03-29T10:54:03.716 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:03.716 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:03.716 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:03.716 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:03.716 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:03.716 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185070150 0 0) 0x7fc9680c8710 con 0x7fc96802f340 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 10 monclient(hunting): my global_id is 4561 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.731+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9680c88a0 con 0x7fc96802f340 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.732+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3363789211 0 0) 0x7fc950004eb0 con 0x7fc968030660 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.732+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9680c8710 con 0x7fc968030660 2024-03-29T10:54:03.717 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.732+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fc960001730 con 0x7fc968030660 2024-03-29T10:54:03.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.732+0000 7fc95effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:03.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.732+0000 7fc95effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:03.718 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.732+0000 7fc95effd640 10 monclient(hunting): dump: 2024-03-29T10:54:03.718 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:03.718 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:03.718 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:03.718 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:03.718 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:03.718 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:03.718 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:03.718 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:03.718 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:03.718 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.732+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3142391489 0 0) 0x7fc9680c88a0 con 0x7fc96802f340 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.732+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc950005ea0 con 0x7fc96802f340 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.732+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fc97004f730 con 0x7fc96802f340 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.732+0000 7fc95effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.732+0000 7fc95effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.732+0000 7fc95effd640 10 monclient(hunting): dump: 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:03.719 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 627146953 0 0) 0x7fc950005ea0 con 0x7fc96802f340 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 >> v1:172.21.15.98:6790/0 conn(0x7fc968030660 legacy=0x7fc96815d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 >> v1:172.21.15.98:6789/0 conn(0x7fc9680389e0 legacy=0x7fc9680c22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc95effd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc95effd640 1 monclient: found mon.b 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc95effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc95effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9680c29b0 con 0x7fc96802f340 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc95effd640 10 monclient: _finish_auth 0 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc95effd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc95effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:03.720 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc96f59e640 5 monclient: authenticate success, global_id 4561 2024-03-29T10:54:03.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc96f59e640 10 monclient: _renew_subs 2024-03-29T10:54:03.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc96f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:54:03.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc96f59e640 1 -- 172.21.15.98:0/919120038 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9680c2c20 con 0x7fc96802f340 2024-03-29T10:54:03.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc96f59e640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:03.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc96f59e640 10 monclient: _renew_subs 2024-03-29T10:54:03.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc96f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:54:03.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc96f59e640 1 -- 172.21.15.98:0/919120038 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9680c6270 con 0x7fc96802f340 2024-03-29T10:54:03.721 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.733+0000 7fc96f59e640 1 librados: init done 2024-03-29T10:54:03.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.738+0000 7fc96f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:03.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.738+0000 7fc96f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:03.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.738+0000 7fc96f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:54:03.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.738+0000 7fc96f59e640 1 -- 172.21.15.98:0/919120038 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc968007b90 con 0x7fc96802f340 2024-03-29T10:54:03.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.738+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc97004fdd0 con 0x7fc96802f340 2024-03-29T10:54:03.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.738+0000 7fc95effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:03.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.738+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fc97004f730 con 0x7fc96802f340 2024-03-29T10:54:03.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.738+0000 7fc95effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:03.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.738+0000 7fc95effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:03.723 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.738+0000 7fc95effd640 10 monclient: dump: 2024-03-29T10:54:03.723 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:03.723 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:03.723 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:03.724 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:03.724 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:03.724 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:03.724 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:03.724 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:03.724 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:03.724 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:03.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.738+0000 7fc96cd99640 4 set_mon_vals no callback set 2024-03-29T10:54:03.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.739+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fc970075710 con 0x7fc96802f340 2024-03-29T10:54:03.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.740+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fc9700ceef0 con 0x7fc96802f340 2024-03-29T10:54:03.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.740+0000 7fc95effd640 10 client.4561.objecter ms_dispatch 0x7fc970000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:03.724 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.740+0000 7fc95effd640 3 client.4561.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:03.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.740+0000 7fc95effd640 3 client.4561.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:03.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.740+0000 7fc95effd640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-29T10:54:03.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.741+0000 7fc96e59c640 10 monclient: get_auth_request con 0x7fc95006e4a0 auth_method 0 2024-03-29T10:54:03.725 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.742+0000 7fc95effd640 10 client.4561.objecter ms_handle_connect 0x7fc95006e4a0 2024-03-29T10:54:03.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.751+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9700cb190 con 0x7fc96802f340 2024-03-29T10:54:03.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.751+0000 7fc95effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:03.732 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:03.751+0000 7fc95effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:04.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.072+0000 7fc96f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:04.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.072+0000 7fc96f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:04.053 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.072+0000 7fc96f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:54:04.054 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.072+0000 7fc96f59e640 1 -- 172.21.15.98:0/919120038 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc968007b90 con 0x7fc96802f340 2024-03-29T10:54:04.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.074+0000 7fc95effd640 1 -- 172.21.15.98:0/919120038 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fc9700cb330 con 0x7fc96802f340 2024-03-29T10:54:04.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.074+0000 7fc95effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:04.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.074+0000 7fc95effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:04.060 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:54:04.060 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:04.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.079+0000 7fc96f59e640 10 librados: watch_flush enter 2024-03-29T10:54:04.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.079+0000 7fc96f59e640 10 librados: watch_flush exit 2024-03-29T10:54:04.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.079+0000 7fc96f59e640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-29T10:54:04.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.079+0000 7fc96f59e640 10 client.4561.objecter successfully canceled tick 2024-03-29T10:54:04.062 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.079+0000 7fc96f59e640 1 -- 172.21.15.98:0/919120038 >> v1:172.21.15.98:6816/1481799426 conn(0x7fc95006e4a0 legacy=0x7fc950070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:04.062 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.079+0000 7fc96f59e640 10 monclient: shutdown 2024-03-29T10:54:04.063 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.079+0000 7fc96f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:04.063 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.079+0000 7fc96f59e640 1 -- 172.21.15.98:0/919120038 >> v1:172.21.15.189:6789/0 conn(0x7fc96802f340 legacy=0x7fc9680c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:04.063 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.081+0000 7fc96f59e640 1 -- 172.21.15.98:0/919120038 shutdown_connections 2024-03-29T10:54:04.063 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.081+0000 7fc96f59e640 1 -- 172.21.15.98:0/919120038 >> 172.21.15.98:0/919120038 conn(0x7fc968005c00 msgr2=0x7fc968034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:04.063 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.081+0000 7fc96f59e640 1 -- 172.21.15.98:0/919120038 shutdown_connections 2024-03-29T10:54:04.063 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.081+0000 7fc96f59e640 1 -- 172.21.15.98:0/919120038 wait complete. 2024-03-29T10:54:04.063 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:04.081+0000 7fc96f59e640 1 librados: shutdown 2024-03-29T10:54:04.079 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:10.886 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:11.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.050+0000 7f3e4f082640 10 monclient: get_monmap_and_config 2024-03-29T10:54:11.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.050+0000 7f3e4f082640 10 monclient: build_initial_monmap 2024-03-29T10:54:11.032 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.050+0000 7f3e4f082640 10 monclient: monmap: 2024-03-29T10:54:11.032 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:11.033 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:11.033 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:54:11.052119+0000 2024-03-29T10:54:11.033 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:54:11.052119+0000 2024-03-29T10:54:11.033 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:11.033 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:11.033 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:11.033 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:54:11.033 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:54:11.033 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:11.034 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.052+0000 7f3e4f082640 1 Processor -- start 2024-03-29T10:54:11.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.052+0000 7f3e4f082640 1 -- start start 2024-03-29T10:54:11.035 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.052+0000 7f3e4f082640 10 monclient: init 2024-03-29T10:54:11.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.054+0000 7f3e4f082640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:11.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.054+0000 7f3e4f082640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:11.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.054+0000 7f3e4f082640 10 monclient(hunting): picked mon.noname-a con 0x7f3e40006a70 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:11.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.054+0000 7f3e4f082640 10 monclient(hunting): picked mon.noname-b con 0x7f3e4002b100 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:11.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.054+0000 7f3e4f082640 10 monclient(hunting): picked mon.noname-c con 0x7f3e40029c30 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:11.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.054+0000 7f3e4f082640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4002a270 con 0x7f3e40006a70 2024-03-29T10:54:11.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.054+0000 7f3e4f082640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4002c440 con 0x7f3e40029c30 2024-03-29T10:54:11.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.054+0000 7f3e4f082640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e400c9440 con 0x7f3e4002b100 2024-03-29T10:54:11.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.054+0000 7f3e4f082640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:11.037 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.054+0000 7f3e4f082640 10 monclient(hunting): authenticate will time out at 1471.252121s 2024-03-29T10:54:11.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.055+0000 7f3e4d87f640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f3e4002b100 0x7f3e400cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:55666/0 (socket says 172.21.15.98:55666) 2024-03-29T10:54:11.038 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.055+0000 7f3e4d87f640 1 -- 172.21.15.98:0/3506672363 learned_addr learned my addr 172.21.15.98:0/3506672363 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:11.039 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.056+0000 7f3e4d07e640 1 -- 172.21.15.98:0/3506672363 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826641017 0 0) 0x7f3e400c9440 con 0x7f3e4002b100 2024-03-29T10:54:11.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.056+0000 7f3e4d07e640 10 monclient(hunting): _negotiate 2024-03-29T10:54:11.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.056+0000 7f3e4d07e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:11.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.056+0000 7f3e4d07e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:11.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.056+0000 7f3e4d07e640 10 monclient(hunting): my global_id is 4613 2024-03-29T10:54:11.040 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.056+0000 7f3e4d07e640 1 -- 172.21.15.98:0/3506672363 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e34003520 con 0x7f3e4002b100 2024-03-29T10:54:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4d07e640 1 -- 172.21.15.98:0/3506672363 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 54867222 0 0) 0x7f3e34003520 con 0x7f3e4002b100 2024-03-29T10:54:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4d07e640 1 -- 172.21.15.98:0/3506672363 >> v1:172.21.15.189:6789/0 conn(0x7f3e40029c30 legacy=0x7f3e400d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4d07e640 1 -- 172.21.15.98:0/3506672363 >> v1:172.21.15.98:6789/0 conn(0x7f3e40006a70 legacy=0x7f3e400c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4d07e640 10 monclient: _finish_hunting 0 2024-03-29T10:54:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4d07e640 1 monclient: found mon.noname-b 2024-03-29T10:54:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4d07e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4d07e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:54:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4d07e640 1 -- 172.21.15.98:0/3506672363 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e400c95d0 con 0x7f3e4002b100 2024-03-29T10:54:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4d07e640 10 monclient: _finish_auth 0 2024-03-29T10:54:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4d07e640 10 monclient: _check_auth_tickets 2024-03-29T10:54:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4d07e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4f082640 5 monclient: authenticate success, global_id 4613 2024-03-29T10:54:11.044 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4f082640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4d07e640 1 -- 172.21.15.98:0/3506672363 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3e44001690 con 0x7f3e4002b100 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.060+0000 7f3e4d07e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.061+0000 7f3e4d07e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.061+0000 7f3e4d07e640 10 monclient: dump: 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:11.045 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.061+0000 7f3e4f082640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:11.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.061+0000 7f3e4d07e640 1 -- 172.21.15.98:0/3506672363 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e440018e0 con 0x7f3e4002b100 2024-03-29T10:54:11.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.061+0000 7f3e4d07e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:11.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.061+0000 7f3e4f082640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:11.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.061+0000 7f3e4f082640 4 set_mon_vals no callback set 2024-03-29T10:54:11.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.061+0000 7f3e4d07e640 1 -- 172.21.15.98:0/3506672363 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3e44001c70 con 0x7f3e4002b100 2024-03-29T10:54:11.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.061+0000 7f3e4d07e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:11.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.061+0000 7f3e4d07e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:11.046 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.061+0000 7f3e4d07e640 10 monclient: dump: 2024-03-29T10:54:11.046 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:11.046 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:11.046 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:11.046 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:11.046 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:11.047 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:11.047 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:11.047 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:11.047 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:11.047 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:11.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.062+0000 7f3e4f082640 10 monclient: shutdown 2024-03-29T10:54:11.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.062+0000 7f3e4f082640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:11.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.062+0000 7f3e4f082640 1 -- 172.21.15.98:0/3506672363 >> v1:172.21.15.98:6790/0 conn(0x7f3e4002b100 legacy=0x7f3e400cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:11.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.062+0000 7f3e4f082640 1 -- 172.21.15.98:0/3506672363 shutdown_connections 2024-03-29T10:54:11.049 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.062+0000 7f3e4f082640 1 -- 172.21.15.98:0/3506672363 >> 172.21.15.98:0/3506672363 conn(0x7f3e400058f0 msgr2=0x7f3e40005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:11.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.062+0000 7f3e4f082640 1 -- 172.21.15.98:0/3506672363 shutdown_connections 2024-03-29T10:54:11.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.062+0000 7f3e4f082640 1 -- 172.21.15.98:0/3506672363 wait complete. 2024-03-29T10:54:11.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.064+0000 7f3e4f082640 10 monclient: build_initial_monmap 2024-03-29T10:54:11.050 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.064+0000 7f3e4f082640 10 monclient: monmap: 2024-03-29T10:54:11.050 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:11.050 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:11.050 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:54:11.050 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:54:11.050 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:11.050 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:11.050 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:11.050 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:54:11.050 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.064+0000 7f3e4f082640 1 librados: starting msgr at 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.064+0000 7f3e4f082640 1 librados: starting objecter 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.064+0000 7f3e4f082640 1 Processor -- start 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.065+0000 7f3e4f082640 1 -- start start 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.065+0000 7f3e4f082640 1 librados: setting wanted keys 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.065+0000 7f3e4f082640 1 librados: calling monclient init 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.065+0000 7f3e4f082640 10 monclient: init 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.065+0000 7f3e4f082640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.065+0000 7f3e4f082640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.065+0000 7f3e4f082640 10 monclient(hunting): picked mon.noname-a con 0x7f3e40006a70 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.065+0000 7f3e4f082640 10 monclient(hunting): picked mon.noname-c con 0x7f3e40029c30 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e4f082640 10 monclient(hunting): picked mon.noname-b con 0x7f3e400caad0 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e4f082640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4002daf0 con 0x7f3e40006a70 2024-03-29T10:54:11.051 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e4f082640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e400caf00 con 0x7f3e400caad0 2024-03-29T10:54:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e4f082640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e400ccbb0 con 0x7f3e40029c30 2024-03-29T10:54:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e4f082640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e4f082640 10 monclient(hunting): authenticate will time out at 1471.264016s 2024-03-29T10:54:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e4d87f640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f3e40029c30 0x7f3e4002d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:58552/0 (socket says 172.21.15.98:58552) 2024-03-29T10:54:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e4d87f640 1 -- 172.21.15.98:0/2980269773 learned_addr learned my addr 172.21.15.98:0/2980269773 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e3effd640 10 client.?.objecter ms_handle_connect 0x7f3e40029c30 2024-03-29T10:54:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673703553 0 0) 0x7f3e400ccbb0 con 0x7f3e40029c30 2024-03-29T10:54:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e3effd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e3effd640 10 monclient(hunting): my global_id is 4616 2024-03-29T10:54:11.052 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.066+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e2c004eb0 con 0x7f3e40029c30 2024-03-29T10:54:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.067+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2980687363 0 0) 0x7f3e2c004eb0 con 0x7f3e40029c30 2024-03-29T10:54:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.067+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e400ccbb0 con 0x7f3e40029c30 2024-03-29T10:54:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.067+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3e44001550 con 0x7f3e40029c30 2024-03-29T10:54:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.067+0000 7f3e3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.067+0000 7f3e3effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:11.056 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.067+0000 7f3e3effd640 10 monclient(hunting): dump: 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.067+0000 7f3e3effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.067+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4224254039 0 0) 0x7f3e400ccbb0 con 0x7f3e40029c30 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.068+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 >> v1:172.21.15.189:6789/0 conn(0x7f3e400caad0 legacy=0x7f3e401664d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:11.057 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 >> v1:172.21.15.98:6789/0 conn(0x7f3e40006a70 legacy=0x7f3e40169b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e3effd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e3effd640 1 monclient: found mon.c 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e3effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4002c8f0 con 0x7f3e40029c30 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e3effd640 10 monclient: _finish_auth 0 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e3effd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e4f082640 5 monclient: authenticate success, global_id 4616 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e4f082640 10 monclient: _renew_subs 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e4f082640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e4f082640 1 -- 172.21.15.98:0/2980269773 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e401692a0 con 0x7f3e40029c30 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.069+0000 7f3e4f082640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:11.058 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.070+0000 7f3e4f082640 10 monclient: _renew_subs 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.070+0000 7f3e4f082640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.070+0000 7f3e4f082640 1 -- 172.21.15.98:0/2980269773 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e40166dc0 con 0x7f3e40029c30 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.070+0000 7f3e4f082640 1 librados: init done 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.070+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e440016b0 con 0x7f3e40029c30 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.070+0000 7f3e3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.070+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3e44004c30 con 0x7f3e40029c30 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.070+0000 7f3e3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.070+0000 7f3e3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.070+0000 7f3e3effd640 10 monclient: dump: 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:11.059 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.070+0000 7f3e3ffff640 4 set_mon_vals no callback set 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.072+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f3e44005e80 con 0x7f3e40029c30 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.072+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f3e44084b70 con 0x7f3e40029c30 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.073+0000 7f3e3effd640 10 client.4616.objecter ms_dispatch 0x7f3e48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.073+0000 7f3e3effd640 3 client.4616.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.073+0000 7f3e3effd640 3 client.4616.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.073+0000 7f3e3effd640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.073+0000 7f3e4e080640 10 monclient: get_auth_request con 0x7f3e2c06dc40 auth_method 0 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.073+0000 7f3e4f082640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.073+0000 7f3e4f082640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:11.060 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.073+0000 7f3e4f082640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:11.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.073+0000 7f3e4f082640 1 -- 172.21.15.98:0/2980269773 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e30005080 con 0x7f3e40029c30 2024-03-29T10:54:11.061 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.073+0000 7f3e3effd640 10 client.4616.objecter ms_handle_connect 0x7f3e2c06dc40 2024-03-29T10:54:11.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.082+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e44050280 con 0x7f3e40029c30 2024-03-29T10:54:11.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.082+0000 7f3e3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:11.064 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.082+0000 7f3e3effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:11.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.411+0000 7f3e4f082640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:11.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.411+0000 7f3e4f082640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:11.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.411+0000 7f3e4f082640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:11.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.411+0000 7f3e4f082640 1 -- 172.21.15.98:0/2980269773 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e30005330 con 0x7f3e40029c30 2024-03-29T10:54:11.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.413+0000 7f3e3effd640 1 -- 172.21.15.98:0/2980269773 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f3e44052380 con 0x7f3e40029c30 2024-03-29T10:54:11.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.414+0000 7f3e3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:11.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.414+0000 7f3e3effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:11.397 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:54:11.397 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:11.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.419+0000 7f3e4f082640 10 librados: watch_flush enter 2024-03-29T10:54:11.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.419+0000 7f3e4f082640 10 librados: watch_flush exit 2024-03-29T10:54:11.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.419+0000 7f3e4f082640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-29T10:54:11.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.419+0000 7f3e4f082640 10 client.4616.objecter successfully canceled tick 2024-03-29T10:54:11.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.419+0000 7f3e4f082640 1 -- 172.21.15.98:0/2980269773 >> v1:172.21.15.98:6816/1481799426 conn(0x7f3e2c06dc40 legacy=0x7f3e2c070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:11.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.419+0000 7f3e4f082640 10 monclient: shutdown 2024-03-29T10:54:11.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.419+0000 7f3e4f082640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:11.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.419+0000 7f3e4f082640 1 -- 172.21.15.98:0/2980269773 >> v1:172.21.15.98:6790/0 conn(0x7f3e40029c30 legacy=0x7f3e4002d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:11.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.419+0000 7f3e4f082640 1 -- 172.21.15.98:0/2980269773 shutdown_connections 2024-03-29T10:54:11.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.419+0000 7f3e4f082640 1 -- 172.21.15.98:0/2980269773 >> 172.21.15.98:0/2980269773 conn(0x7f3e400058f0 msgr2=0x7f3e40032d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:11.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.419+0000 7f3e4f082640 1 -- 172.21.15.98:0/2980269773 shutdown_connections 2024-03-29T10:54:11.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.419+0000 7f3e4f082640 1 -- 172.21.15.98:0/2980269773 wait complete. 2024-03-29T10:54:11.403 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:11.419+0000 7f3e4f082640 1 librados: shutdown 2024-03-29T10:54:11.418 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:18.225 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:18.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.388+0000 7fd026d4d640 10 monclient: get_monmap_and_config 2024-03-29T10:54:18.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.388+0000 7fd026d4d640 10 monclient: build_initial_monmap 2024-03-29T10:54:18.372 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.388+0000 7fd026d4d640 10 monclient: monmap: 2024-03-29T10:54:18.372 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:18.372 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:18.372 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:54:18.390555+0000 2024-03-29T10:54:18.372 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:54:18.390555+0000 2024-03-29T10:54:18.372 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:18.372 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:18.372 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:18.372 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:54:18.372 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:54:18.373 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:18.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.391+0000 7fd026d4d640 1 Processor -- start 2024-03-29T10:54:18.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.391+0000 7fd026d4d640 1 -- start start 2024-03-29T10:54:18.373 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.391+0000 7fd026d4d640 10 monclient: init 2024-03-29T10:54:18.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.392+0000 7fd026d4d640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:18.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.392+0000 7fd026d4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:18.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.392+0000 7fd026d4d640 10 monclient(hunting): picked mon.noname-a con 0x7fd018034850 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:18.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.392+0000 7fd026d4d640 10 monclient(hunting): picked mon.noname-c con 0x7fd0180c9550 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:18.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.392+0000 7fd026d4d640 10 monclient(hunting): picked mon.noname-b con 0x7fd01802aac0 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:18.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.392+0000 7fd026d4d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01802c440 con 0x7fd018034850 2024-03-29T10:54:18.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.392+0000 7fd026d4d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01802aed0 con 0x7fd0180c9550 2024-03-29T10:54:18.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.392+0000 7fd026d4d640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01802b190 con 0x7fd01802aac0 2024-03-29T10:54:18.375 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.392+0000 7fd026d4d640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:18.376 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.392+0000 7fd026d4d640 10 monclient(hunting): authenticate will time out at 1478.590903s 2024-03-29T10:54:18.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.396+0000 7fd02654c640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fd01802aac0 0x7fd0180cc5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:40324/0 (socket says 172.21.15.98:40324) 2024-03-29T10:54:18.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.396+0000 7fd02654c640 1 -- 172.21.15.98:0/3559363189 learned_addr learned my addr 172.21.15.98:0/3559363189 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:18.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.396+0000 7fd024d49640 1 -- 172.21.15.98:0/3559363189 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746530980 0 0) 0x7fd01802b190 con 0x7fd01802aac0 2024-03-29T10:54:18.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.396+0000 7fd024d49640 10 monclient(hunting): _negotiate 2024-03-29T10:54:18.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.396+0000 7fd024d49640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:18.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.396+0000 7fd024d49640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:18.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.396+0000 7fd024d49640 10 monclient(hunting): my global_id is 4619 2024-03-29T10:54:18.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.396+0000 7fd024d49640 1 -- 172.21.15.98:0/3559363189 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd00c003520 con 0x7fd01802aac0 2024-03-29T10:54:18.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 1 -- 172.21.15.98:0/3559363189 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 393682679 0 0) 0x7fd00c003520 con 0x7fd01802aac0 2024-03-29T10:54:18.379 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 1 -- 172.21.15.98:0/3559363189 >> v1:172.21.15.189:6789/0 conn(0x7fd0180c9550 legacy=0x7fd0180cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:18.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 1 -- 172.21.15.98:0/3559363189 >> v1:172.21.15.98:6789/0 conn(0x7fd018034850 legacy=0x7fd0180c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:18.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 10 monclient: _finish_hunting 0 2024-03-29T10:54:18.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 1 monclient: found mon.noname-b 2024-03-29T10:54:18.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:18.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:54:18.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 1 -- 172.21.15.98:0/3559363189 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd018033060 con 0x7fd01802aac0 2024-03-29T10:54:18.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 10 monclient: _finish_auth 0 2024-03-29T10:54:18.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 10 monclient: _check_auth_tickets 2024-03-29T10:54:18.380 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 1 -- 172.21.15.98:0/3559363189 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd01c001690 con 0x7fd01802aac0 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 10 monclient: dump: 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:18.381 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 1 -- 172.21.15.98:0/3559363189 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd01c001860 con 0x7fd01802aac0 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 1 -- 172.21.15.98:0/3559363189 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd01c001cb0 con 0x7fd01802aac0 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd024d49640 10 monclient: dump: 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:18.382 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:18.383 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:18.383 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd026d4d640 5 monclient: authenticate success, global_id 4619 2024-03-29T10:54:18.383 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd026d4d640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:18.383 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.397+0000 7fd026d4d640 4 set_mon_vals no callback set 2024-03-29T10:54:18.383 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.398+0000 7fd026d4d640 10 monclient: shutdown 2024-03-29T10:54:18.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.398+0000 7fd026d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:18.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.398+0000 7fd026d4d640 1 -- 172.21.15.98:0/3559363189 >> v1:172.21.15.98:6790/0 conn(0x7fd01802aac0 legacy=0x7fd0180cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:18.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.399+0000 7fd026d4d640 1 -- 172.21.15.98:0/3559363189 shutdown_connections 2024-03-29T10:54:18.384 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.399+0000 7fd026d4d640 1 -- 172.21.15.98:0/3559363189 >> 172.21.15.98:0/3559363189 conn(0x7fd0180058f0 msgr2=0x7fd018005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:18.385 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.403+0000 7fd026d4d640 1 -- 172.21.15.98:0/3559363189 shutdown_connections 2024-03-29T10:54:18.389 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.407+0000 7fd026d4d640 1 -- 172.21.15.98:0/3559363189 wait complete. 2024-03-29T10:54:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.408+0000 7fd026d4d640 10 monclient: build_initial_monmap 2024-03-29T10:54:18.392 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.408+0000 7fd026d4d640 10 monclient: monmap: 2024-03-29T10:54:18.392 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:18.392 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:18.392 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:54:18.392 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:54:18.392 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:18.392 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:18.392 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:18.393 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:54:18.393 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:54:18.393 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.408+0000 7fd026d4d640 1 librados: starting msgr at 2024-03-29T10:54:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.408+0000 7fd026d4d640 1 librados: starting objecter 2024-03-29T10:54:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.408+0000 7fd026d4d640 1 Processor -- start 2024-03-29T10:54:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.408+0000 7fd026d4d640 1 -- start start 2024-03-29T10:54:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.408+0000 7fd026d4d640 1 librados: setting wanted keys 2024-03-29T10:54:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.408+0000 7fd026d4d640 1 librados: calling monclient init 2024-03-29T10:54:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.408+0000 7fd026d4d640 10 monclient: init 2024-03-29T10:54:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.409+0000 7fd026d4d640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:18.393 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.409+0000 7fd026d4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:18.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.409+0000 7fd026d4d640 10 monclient(hunting): picked mon.noname-c con 0x7fd01802aac0 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:18.394 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.409+0000 7fd026d4d640 10 monclient(hunting): picked mon.noname-a con 0x7fd018034850 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:18.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.411+0000 7fd026d4d640 10 monclient(hunting): picked mon.noname-b con 0x7fd0180c9550 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:18.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.411+0000 7fd026d4d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd018164450 con 0x7fd018034850 2024-03-29T10:54:18.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.411+0000 7fd026d4d640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd018164ff0 con 0x7fd0180c9550 2024-03-29T10:54:18.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.411+0000 7fd026d4d640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd018164090 con 0x7fd01802aac0 2024-03-29T10:54:18.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.411+0000 7fd026d4d640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:18.395 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.411+0000 7fd026d4d640 10 monclient(hunting): authenticate will time out at 1478.609216s 2024-03-29T10:54:18.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.413+0000 7fd025d4b640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fd01802aac0 0x7fd0180bf7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:41082/0 (socket says 172.21.15.98:41082) 2024-03-29T10:54:18.396 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.413+0000 7fd025d4b640 1 -- 172.21.15.98:0/2456293499 learned_addr learned my addr 172.21.15.98:0/2456293499 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:18.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.414+0000 7fd0167fc640 10 client.?.objecter ms_handle_connect 0x7fd01802aac0 2024-03-29T10:54:18.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.414+0000 7fd0167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:18.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.414+0000 7fd0167fc640 10 client.?.objecter ms_handle_connect 0x7fd0180c9550 2024-03-29T10:54:18.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.414+0000 7fd0167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:18.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.414+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688771925 0 0) 0x7fd018164ff0 con 0x7fd0180c9550 2024-03-29T10:54:18.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.414+0000 7fd0167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:18.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.414+0000 7fd0167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:18.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.414+0000 7fd0167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:18.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.414+0000 7fd0167fc640 10 monclient(hunting): my global_id is 4567 2024-03-29T10:54:18.397 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.414+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd004004eb0 con 0x7fd0180c9550 2024-03-29T10:54:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.415+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 90704078 0 0) 0x7fd004004eb0 con 0x7fd0180c9550 2024-03-29T10:54:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.415+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd018164ff0 con 0x7fd0180c9550 2024-03-29T10:54:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.415+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd01c001b70 con 0x7fd0180c9550 2024-03-29T10:54:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.415+0000 7fd0167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.415+0000 7fd0167fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:18.398 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.415+0000 7fd0167fc640 10 monclient(hunting): dump: 2024-03-29T10:54:18.398 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:18.398 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:18.398 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:18.398 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:18.398 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:18.398 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:18.399 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:18.399 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:18.399 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:18.399 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:18.399 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.415+0000 7fd0167fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:18.399 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 <== mon.1 v1:172.21.15.189:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3782142529 0 0) 0x7fd018164ff0 con 0x7fd0180c9550 2024-03-29T10:54:18.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 >> v1:172.21.15.98:6790/0 conn(0x7fd01802aac0 legacy=0x7fd0180bf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:18.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 >> v1:172.21.15.98:6789/0 conn(0x7fd018034850 legacy=0x7fd0181648e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:18.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:18.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 1 monclient: found mon.b 2024-03-29T10:54:18.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:18.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:54:18.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01815e6f0 con 0x7fd0180c9550 2024-03-29T10:54:18.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:18.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:18.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:18.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 <== mon.1 v1:172.21.15.189:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd01c004bf0 con 0x7fd0180c9550 2024-03-29T10:54:18.400 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 <== mon.1 v1:172.21.15.189:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fd01c001b70 con 0x7fd0180c9550 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.416+0000 7fd0167fc640 10 monclient: dump: 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:18.401 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.417+0000 7fd017fff640 4 set_mon_vals no callback set 2024-03-29T10:54:18.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.418+0000 7fd026d4d640 5 monclient: authenticate success, global_id 4567 2024-03-29T10:54:18.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.418+0000 7fd026d4d640 10 monclient: _renew_subs 2024-03-29T10:54:18.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.418+0000 7fd026d4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:54:18.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.418+0000 7fd026d4d640 1 -- 172.21.15.98:0/2456293499 --> v1:172.21.15.189:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01815d7e0 con 0x7fd0180c9550 2024-03-29T10:54:18.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.418+0000 7fd026d4d640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:18.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.418+0000 7fd026d4d640 10 monclient: _renew_subs 2024-03-29T10:54:18.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.418+0000 7fd026d4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:54:18.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.418+0000 7fd026d4d640 1 -- 172.21.15.98:0/2456293499 --> v1:172.21.15.189:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0180c04c0 con 0x7fd0180c9550 2024-03-29T10:54:18.402 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.418+0000 7fd026d4d640 1 librados: init done 2024-03-29T10:54:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.422+0000 7fd026d4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.422+0000 7fd026d4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.422+0000 7fd026d4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:54:18.406 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.422+0000 7fd026d4d640 1 -- 172.21.15.98:0/2456293499 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd018164220 con 0x7fd0180c9550 2024-03-29T10:54:18.408 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.426+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 <== mon.1 v1:172.21.15.189:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fd01c01ac50 con 0x7fd0180c9550 2024-03-29T10:54:18.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.426+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 <== mon.1 v1:172.21.15.189:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fd01c0846b0 con 0x7fd0180c9550 2024-03-29T10:54:18.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.426+0000 7fd0167fc640 10 client.4567.objecter ms_dispatch 0x7fd020000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:18.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.426+0000 7fd0167fc640 3 client.4567.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:18.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.426+0000 7fd0167fc640 3 client.4567.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:18.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.426+0000 7fd0167fc640 20 client.4567.objecter dump_active .. 0 homeless 2024-03-29T10:54:18.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.427+0000 7fd025d4b640 10 monclient: get_auth_request con 0x7fd00406e010 auth_method 0 2024-03-29T10:54:18.409 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.427+0000 7fd0167fc640 10 client.4567.objecter ms_handle_connect 0x7fd00406e010 2024-03-29T10:54:18.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.430+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 <== mon.1 v1:172.21.15.189:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd01c04ff90 con 0x7fd0180c9550 2024-03-29T10:54:18.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.430+0000 7fd0167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:18.412 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.430+0000 7fd0167fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:18.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.794+0000 7fd026d4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:18.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.794+0000 7fd026d4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:18.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.794+0000 7fd026d4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.189:6789/0 2024-03-29T10:54:18.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.794+0000 7fd026d4d640 1 -- 172.21.15.98:0/2456293499 --> v1:172.21.15.189:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0180be070 con 0x7fd0180c9550 2024-03-29T10:54:18.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.794+0000 7fd0167fc640 1 -- 172.21.15.98:0/2456293499 <== mon.1 v1:172.21.15.189:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fd01c04ff90 con 0x7fd0180c9550 2024-03-29T10:54:18.776 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.794+0000 7fd0167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:18.777 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.794+0000 7fd0167fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:18.777 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:54:18.777 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:18.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.798+0000 7fd026d4d640 10 librados: watch_flush enter 2024-03-29T10:54:18.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.801+0000 7fd026d4d640 10 librados: watch_flush exit 2024-03-29T10:54:18.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.801+0000 7fd026d4d640 20 client.4567.objecter shutdown clearing up homeless session... 2024-03-29T10:54:18.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.801+0000 7fd026d4d640 10 client.4567.objecter successfully canceled tick 2024-03-29T10:54:18.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.801+0000 7fd026d4d640 1 -- 172.21.15.98:0/2456293499 >> v1:172.21.15.98:6816/1481799426 conn(0x7fd00406e010 legacy=0x7fd0040704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:18.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.801+0000 7fd026d4d640 10 monclient: shutdown 2024-03-29T10:54:18.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.801+0000 7fd026d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:18.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.801+0000 7fd026d4d640 1 -- 172.21.15.98:0/2456293499 >> v1:172.21.15.189:6789/0 conn(0x7fd0180c9550 legacy=0x7fd01815dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:18.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.802+0000 7fd026d4d640 1 -- 172.21.15.98:0/2456293499 shutdown_connections 2024-03-29T10:54:18.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.802+0000 7fd026d4d640 1 -- 172.21.15.98:0/2456293499 >> 172.21.15.98:0/2456293499 conn(0x7fd0180058f0 msgr2=0x7fd018029610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:18.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.803+0000 7fd026d4d640 1 -- 172.21.15.98:0/2456293499 shutdown_connections 2024-03-29T10:54:18.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.803+0000 7fd026d4d640 1 -- 172.21.15.98:0/2456293499 wait complete. 2024-03-29T10:54:18.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:18.803+0000 7fd026d4d640 1 librados: shutdown 2024-03-29T10:54:18.803 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:25.611 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:25.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.778+0000 7f5fe70f6640 10 monclient: get_monmap_and_config 2024-03-29T10:54:25.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.778+0000 7f5fe70f6640 10 monclient: build_initial_monmap 2024-03-29T10:54:25.760 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.778+0000 7f5fe70f6640 10 monclient: monmap: 2024-03-29T10:54:25.760 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:25.760 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:25.760 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:54:25.779518+0000 2024-03-29T10:54:25.760 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:54:25.779518+0000 2024-03-29T10:54:25.760 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:25.760 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:25.760 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:25.760 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:54:25.760 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:54:25.761 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:25.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.779+0000 7f5fe70f6640 1 Processor -- start 2024-03-29T10:54:25.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.779+0000 7f5fe70f6640 1 -- start start 2024-03-29T10:54:25.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.779+0000 7f5fe70f6640 10 monclient: init 2024-03-29T10:54:25.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.780+0000 7f5fe70f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:25.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.780+0000 7f5fe70f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:25.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.780+0000 7f5fe70f6640 10 monclient(hunting): picked mon.noname-a con 0x7f5fe00b0860 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:25.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.780+0000 7f5fe70f6640 10 monclient(hunting): picked mon.noname-c con 0x7f5fe013ef30 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:25.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.780+0000 7f5fe70f6640 10 monclient(hunting): picked mon.noname-b con 0x7f5fe00af8e0 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:25.762 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.780+0000 7f5fe70f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe00b0c40 con 0x7f5fe00b0860 2024-03-29T10:54:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.780+0000 7f5fe70f6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe00aed10 con 0x7f5fe013ef30 2024-03-29T10:54:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.780+0000 7f5fe70f6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe00afd50 con 0x7f5fe00af8e0 2024-03-29T10:54:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.780+0000 7f5fe70f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.780+0000 7f5fe70f6640 10 monclient(hunting): authenticate will time out at 1485.977808s 2024-03-29T10:54:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.781+0000 7f5fe4e92640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5fe00b0860 0x7f5fe013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46474/0 (socket says 172.21.15.98:46474) 2024-03-29T10:54:25.763 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.781+0000 7f5fe4e92640 1 -- 172.21.15.98:0/1388260223 learned_addr learned my addr 172.21.15.98:0/1388260223 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.781+0000 7f5fdf7fe640 1 -- 172.21.15.98:0/1388260223 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895959206 0 0) 0x7f5fe00b0c40 con 0x7f5fe00b0860 2024-03-29T10:54:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.781+0000 7f5fdf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.781+0000 7f5fdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.781+0000 7f5fdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.781+0000 7f5fdf7fe640 10 monclient(hunting): my global_id is 4560 2024-03-29T10:54:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.781+0000 7f5fdf7fe640 1 -- 172.21.15.98:0/1388260223 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc8003520 con 0x7f5fe00b0860 2024-03-29T10:54:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 1 -- 172.21.15.98:0/1388260223 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196236651 0 0) 0x7f5fe00afd50 con 0x7f5fe00af8e0 2024-03-29T10:54:25.766 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 10 monclient(hunting): my global_id is 4625 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 1 -- 172.21.15.98:0/1388260223 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe00b0c40 con 0x7f5fe00af8e0 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 1 -- 172.21.15.98:0/1388260223 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1842187308 0 0) 0x7f5fc8003520 con 0x7f5fe00b0860 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 1 -- 172.21.15.98:0/1388260223 >> v1:172.21.15.98:6790/0 conn(0x7f5fe00af8e0 legacy=0x7f5fe0149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 1 -- 172.21.15.98:0/1388260223 >> v1:172.21.15.189:6789/0 conn(0x7f5fe013ef30 legacy=0x7f5fe00af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 1 monclient: found mon.noname-a 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 1 -- 172.21.15.98:0/1388260223 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe00ab1b0 con 0x7f5fe00b0860 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:25.767 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fe70f6640 5 monclient: authenticate success, global_id 4560 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fe70f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 1 -- 172.21.15.98:0/1388260223 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5fd0004c90 con 0x7f5fe00b0860 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 10 monclient: dump: 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:25.768 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fe70f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 1 -- 172.21.15.98:0/1388260223 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fd0004eb0 con 0x7f5fe00b0860 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fdf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fe70f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.782+0000 7f5fe70f6640 4 set_mon_vals no callback set 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.783+0000 7f5fdf7fe640 1 -- 172.21.15.98:0/1388260223 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5fd0001410 con 0x7f5fe00b0860 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.783+0000 7f5fdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.783+0000 7f5fdf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.783+0000 7f5fdf7fe640 10 monclient: dump: 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:25.769 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.783+0000 7f5fe70f6640 10 monclient: shutdown 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.783+0000 7f5fe70f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.783+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1388260223 >> v1:172.21.15.98:6789/0 conn(0x7f5fe00b0860 legacy=0x7f5fe013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.783+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1388260223 shutdown_connections 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.783+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1388260223 >> 172.21.15.98:0/1388260223 conn(0x7f5fe006b660 msgr2=0x7f5fe006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.783+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1388260223 shutdown_connections 2024-03-29T10:54:25.770 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.783+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1388260223 wait complete. 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.784+0000 7f5fe70f6640 10 monclient: build_initial_monmap 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.784+0000 7f5fe70f6640 10 monclient: monmap: 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.785+0000 7f5fe70f6640 1 librados: starting msgr at 2024-03-29T10:54:25.771 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.785+0000 7f5fe70f6640 1 librados: starting objecter 2024-03-29T10:54:25.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.785+0000 7f5fe70f6640 1 Processor -- start 2024-03-29T10:54:25.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.785+0000 7f5fe70f6640 1 -- start start 2024-03-29T10:54:25.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.785+0000 7f5fe70f6640 1 librados: setting wanted keys 2024-03-29T10:54:25.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.785+0000 7f5fe70f6640 1 librados: calling monclient init 2024-03-29T10:54:25.772 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.785+0000 7f5fe70f6640 10 monclient: init 2024-03-29T10:54:25.778 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.786+0000 7f5fe70f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.786+0000 7f5fe70f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.786+0000 7f5fe70f6640 10 monclient(hunting): picked mon.noname-c con 0x7f5fe00af8e0 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.786+0000 7f5fe70f6640 10 monclient(hunting): picked mon.noname-a con 0x7f5fe00b0860 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.786+0000 7f5fe70f6640 10 monclient(hunting): picked mon.noname-b con 0x7f5fe013ef30 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.786+0000 7f5fe70f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe00b3270 con 0x7f5fe00b0860 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.786+0000 7f5fe70f6640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe01deb20 con 0x7f5fe013ef30 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.786+0000 7f5fe70f6640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe00b3ff0 con 0x7f5fe00af8e0 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.786+0000 7f5fe70f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.786+0000 7f5fe70f6640 10 monclient(hunting): authenticate will time out at 1485.983678s 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.786+0000 7f5fdffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5fe00b0860 0x7f5fe01ddd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46740/0 (socket says 172.21.15.98:46740) 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.786+0000 7f5fdffff640 1 -- 172.21.15.98:0/1147637719 learned_addr learned my addr 172.21.15.98:0/1147637719 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 client.?.objecter ms_handle_connect 0x7f5fe00b0860 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:25.779 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661521358 0 0) 0x7f5fe00b3270 con 0x7f5fe00b0860 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 monclient(hunting): my global_id is 4563 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fcc004eb0 con 0x7f5fe00b0860 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 client.?.objecter ms_handle_connect 0x7f5fe00af8e0 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6897763 0 0) 0x7f5fe00b3ff0 con 0x7f5fe00af8e0 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 monclient(hunting): my global_id is 4628 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe00b3270 con 0x7f5fe00af8e0 2024-03-29T10:54:25.780 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 client.?.objecter ms_handle_connect 0x7f5fe013ef30 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.787+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 628389240 0 0) 0x7f5fcc004eb0 con 0x7f5fe00b0860 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fe00b3ff0 con 0x7f5fe00b0860 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5fd40018b0 con 0x7f5fe00b0860 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient(hunting): dump: 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:25.781 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:25.782 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:25.782 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:25.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:25.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2292867328 0 0) 0x7f5fe00b3270 con 0x7f5fe00af8e0 2024-03-29T10:54:25.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fcc004eb0 con 0x7f5fe00af8e0 2024-03-29T10:54:25.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5fd0004c10 con 0x7f5fe00af8e0 2024-03-29T10:54:25.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:25.782 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient(hunting): dump: 2024-03-29T10:54:25.782 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:25.782 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:25.782 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:25.782 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879574262 0 0) 0x7f5fe01deb20 con 0x7f5fe013ef30 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient(hunting): my global_id is 4573 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe00b3270 con 0x7f5fe013ef30 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2396542579 0 0) 0x7f5fe00b3ff0 con 0x7f5fe00b0860 2024-03-29T10:54:25.783 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 >> v1:172.21.15.98:6790/0 conn(0x7f5fe00af8e0 legacy=0x7f5fe00b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 >> v1:172.21.15.189:6789/0 conn(0x7f5fe013ef30 legacy=0x7f5fe01de410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:54:25.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 1 monclient: found mon.a 2024-03-29T10:54:25.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:25.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:25.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe00b42d0 con 0x7f5fe00b0860 2024-03-29T10:54:25.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient: _finish_auth 0 2024-03-29T10:54:25.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:54:25.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.788+0000 7f5fdd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:25.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fe70f6640 5 monclient: authenticate success, global_id 4563 2024-03-29T10:54:25.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fe70f6640 10 monclient: _renew_subs 2024-03-29T10:54:25.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fe70f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:25.784 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1147637719 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe0135820 con 0x7f5fe00b0860 2024-03-29T10:54:25.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fe70f6640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:25.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fe70f6640 10 monclient: _renew_subs 2024-03-29T10:54:25.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fe70f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:25.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1147637719 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe00b2f70 con 0x7f5fe00b0860 2024-03-29T10:54:25.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fe70f6640 1 librados: init done 2024-03-29T10:54:25.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fd4003610 con 0x7f5fe00b0860 2024-03-29T10:54:25.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fdd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:25.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5fd40039e0 con 0x7f5fe00b0860 2024-03-29T10:54:25.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fdd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fdd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:25.785 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fdd7fa640 10 monclient: dump: 2024-03-29T10:54:25.785 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:25.785 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:25.786 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:25.786 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:25.786 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.786 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:25.786 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:25.786 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:25.786 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:25.786 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:25.786 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.789+0000 7f5fdeffd640 4 set_mon_vals no callback set 2024-03-29T10:54:25.786 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.790+0000 7f5fe70f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:25.786 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.790+0000 7f5fe70f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:25.786 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.790+0000 7f5fe70f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:25.786 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.790+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1147637719 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc0005080 con 0x7f5fe00b0860 2024-03-29T10:54:25.787 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.792+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f5fd4003b40 con 0x7f5fe00b0860 2024-03-29T10:54:25.787 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.793+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7f5fd4084830 con 0x7f5fe00b0860 2024-03-29T10:54:25.787 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.793+0000 7f5fdd7fa640 10 client.4563.objecter ms_dispatch 0x7f5fe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:25.787 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.793+0000 7f5fdd7fa640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:25.787 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.793+0000 7f5fdd7fa640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:25.787 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.793+0000 7f5fdd7fa640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-29T10:54:25.787 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.793+0000 7f5fe4e92640 10 monclient: get_auth_request con 0x7f5fcc06dfe0 auth_method 0 2024-03-29T10:54:25.787 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.793+0000 7f5fdd7fa640 10 client.4563.objecter ms_handle_connect 0x7f5fcc06dfe0 2024-03-29T10:54:25.788 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.798+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5fd4050110 con 0x7f5fe00b0860 2024-03-29T10:54:25.788 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.798+0000 7f5fdd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:25.788 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:25.798+0000 7f5fdd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:26.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.164+0000 7f5fe70f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:26.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.164+0000 7f5fe70f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:26.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.164+0000 7f5fe70f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:26.146 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.164+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1147637719 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fc0005330 con 0x7f5fe00b0860 2024-03-29T10:54:26.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.166+0000 7f5fdd7fa640 1 -- 172.21.15.98:0/1147637719 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7f5fd4052040 con 0x7f5fe00b0860 2024-03-29T10:54:26.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.166+0000 7f5fdd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:26.147 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.166+0000 7f5fdd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:26.148 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:54:26.148 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:26.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.170+0000 7f5fe70f6640 10 librados: watch_flush enter 2024-03-29T10:54:26.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.170+0000 7f5fe70f6640 10 librados: watch_flush exit 2024-03-29T10:54:26.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.170+0000 7f5fe70f6640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-29T10:54:26.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.170+0000 7f5fe70f6640 10 client.4563.objecter successfully canceled tick 2024-03-29T10:54:26.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.170+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1147637719 >> v1:172.21.15.98:6816/1481799426 conn(0x7f5fcc06dfe0 legacy=0x7f5fcc0704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:26.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.170+0000 7f5fe70f6640 10 monclient: shutdown 2024-03-29T10:54:26.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.170+0000 7f5fe70f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:26.152 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.170+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1147637719 >> v1:172.21.15.98:6789/0 conn(0x7f5fe00b0860 legacy=0x7f5fe01ddd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:26.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.171+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1147637719 shutdown_connections 2024-03-29T10:54:26.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.171+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1147637719 >> 172.21.15.98:0/1147637719 conn(0x7f5fe006b660 msgr2=0x7f5fe01c9250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:26.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.171+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1147637719 shutdown_connections 2024-03-29T10:54:26.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.171+0000 7f5fe70f6640 1 -- 172.21.15.98:0/1147637719 wait complete. 2024-03-29T10:54:26.153 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:26.171+0000 7f5fe70f6640 1 librados: shutdown 2024-03-29T10:54:26.166 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:32.974 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:33.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.135+0000 7fdb13286640 10 monclient: get_monmap_and_config 2024-03-29T10:54:33.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.135+0000 7fdb13286640 10 monclient: build_initial_monmap 2024-03-29T10:54:33.116 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.135+0000 7fdb13286640 10 monclient: monmap: 2024-03-29T10:54:33.116 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:33.117 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:33.117 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:54:33.136089+0000 2024-03-29T10:54:33.117 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:54:33.136089+0000 2024-03-29T10:54:33.117 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:33.117 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:33.117 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:33.117 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:54:33.117 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:54:33.117 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:33.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.136+0000 7fdb13286640 1 Processor -- start 2024-03-29T10:54:33.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.136+0000 7fdb13286640 1 -- start start 2024-03-29T10:54:33.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.136+0000 7fdb13286640 10 monclient: init 2024-03-29T10:54:33.122 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.137+0000 7fdb13286640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:33.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.137+0000 7fdb13286640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:33.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.137+0000 7fdb13286640 10 monclient(hunting): picked mon.noname-c con 0x7fdb0c0b0860 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:33.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.137+0000 7fdb13286640 10 monclient(hunting): picked mon.noname-b con 0x7fdb0c13ef30 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:33.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.137+0000 7fdb13286640 10 monclient(hunting): picked mon.noname-a con 0x7fdb0c0af8e0 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:33.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.137+0000 7fdb13286640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0c0b0c40 con 0x7fdb0c0af8e0 2024-03-29T10:54:33.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.137+0000 7fdb13286640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0c0aed10 con 0x7fdb0c0b0860 2024-03-29T10:54:33.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.137+0000 7fdb13286640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0c0afd50 con 0x7fdb0c13ef30 2024-03-29T10:54:33.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.137+0000 7fdb13286640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:33.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.137+0000 7fdb13286640 10 monclient(hunting): authenticate will time out at 1493.334658s 2024-03-29T10:54:33.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.138+0000 7fdb11823640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdb0c0af8e0 0x7fdb0c149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:45682/0 (socket says 172.21.15.98:45682) 2024-03-29T10:54:33.123 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.138+0000 7fdb11823640 1 -- 172.21.15.98:0/3753531556 learned_addr learned my addr 172.21.15.98:0/3753531556 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:33.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.138+0000 7fdb03fff640 1 -- 172.21.15.98:0/3753531556 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322422964 0 0) 0x7fdb0c0b0c40 con 0x7fdb0c0af8e0 2024-03-29T10:54:33.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.138+0000 7fdb03fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:33.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.138+0000 7fdb03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:33.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.138+0000 7fdb03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:33.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.138+0000 7fdb03fff640 10 monclient(hunting): my global_id is 4566 2024-03-29T10:54:33.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.138+0000 7fdb03fff640 1 -- 172.21.15.98:0/3753531556 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaf8003520 con 0x7fdb0c0af8e0 2024-03-29T10:54:33.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.138+0000 7fdb03fff640 1 -- 172.21.15.98:0/3753531556 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039634256 0 0) 0x7fdb0c0aed10 con 0x7fdb0c0b0860 2024-03-29T10:54:33.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.138+0000 7fdb03fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:33.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.138+0000 7fdb03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:33.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.138+0000 7fdb03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:33.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.139+0000 7fdb03fff640 10 monclient(hunting): my global_id is 4576 2024-03-29T10:54:33.124 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.139+0000 7fdb03fff640 1 -- 172.21.15.98:0/3753531556 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb0c0b0c40 con 0x7fdb0c0b0860 2024-03-29T10:54:33.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.139+0000 7fdb03fff640 1 -- 172.21.15.98:0/3753531556 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3286466135 0 0) 0x7fdaf8003520 con 0x7fdb0c0af8e0 2024-03-29T10:54:33.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.139+0000 7fdb03fff640 1 -- 172.21.15.98:0/3753531556 >> v1:172.21.15.98:6790/0 conn(0x7fdb0c13ef30 legacy=0x7fdb0c0af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:33.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 1 -- 172.21.15.98:0/3753531556 >> v1:172.21.15.189:6789/0 conn(0x7fdb0c0b0860 legacy=0x7fdb0c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:33.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:33.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 1 monclient: found mon.noname-a 2024-03-29T10:54:33.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:33.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:54:33.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 1 -- 172.21.15.98:0/3753531556 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c0ab1b0 con 0x7fdb0c0af8e0 2024-03-29T10:54:33.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 10 monclient: _finish_auth 0 2024-03-29T10:54:33.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:33.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:33.125 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 1 -- 172.21.15.98:0/3753531556 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fdb04001a10 con 0x7fdb0c0af8e0 2024-03-29T10:54:33.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:33.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:33.126 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 10 monclient: dump: 2024-03-29T10:54:33.126 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:33.126 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:33.126 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:33.126 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:33.126 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:33.126 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:33.126 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:33.126 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:33.126 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 1 -- 172.21.15.98:0/3753531556 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb04001b70 con 0x7fdb0c0af8e0 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 1 -- 172.21.15.98:0/3753531556 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fdb04001f80 con 0x7fdb0c0af8e0 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb03fff640 10 monclient: dump: 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:33.127 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:33.128 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:33.128 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:33.128 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:33.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb13286640 5 monclient: authenticate success, global_id 4566 2024-03-29T10:54:33.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb13286640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:33.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.140+0000 7fdb13286640 4 set_mon_vals no callback set 2024-03-29T10:54:33.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.141+0000 7fdb13286640 10 monclient: shutdown 2024-03-29T10:54:33.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.141+0000 7fdb13286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:33.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.141+0000 7fdb13286640 1 -- 172.21.15.98:0/3753531556 >> v1:172.21.15.98:6789/0 conn(0x7fdb0c0af8e0 legacy=0x7fdb0c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:33.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.141+0000 7fdb13286640 1 -- 172.21.15.98:0/3753531556 shutdown_connections 2024-03-29T10:54:33.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.141+0000 7fdb13286640 1 -- 172.21.15.98:0/3753531556 >> 172.21.15.98:0/3753531556 conn(0x7fdb0c06b660 msgr2=0x7fdb0c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:33.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.141+0000 7fdb13286640 1 -- 172.21.15.98:0/3753531556 shutdown_connections 2024-03-29T10:54:33.128 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.141+0000 7fdb13286640 1 -- 172.21.15.98:0/3753531556 wait complete. 2024-03-29T10:54:33.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.142+0000 7fdb13286640 10 monclient: build_initial_monmap 2024-03-29T10:54:33.134 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.142+0000 7fdb13286640 10 monclient: monmap: 2024-03-29T10:54:33.134 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:33.134 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:33.134 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:54:33.134 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:54:33.134 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:33.134 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:33.135 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:33.135 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:54:33.135 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:54:33.135 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:33.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.142+0000 7fdb13286640 1 librados: starting msgr at 2024-03-29T10:54:33.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.142+0000 7fdb13286640 1 librados: starting objecter 2024-03-29T10:54:33.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.142+0000 7fdb13286640 1 Processor -- start 2024-03-29T10:54:33.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 1 -- start start 2024-03-29T10:54:33.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 1 librados: setting wanted keys 2024-03-29T10:54:33.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 1 librados: calling monclient init 2024-03-29T10:54:33.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 10 monclient: init 2024-03-29T10:54:33.135 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 10 monclient(hunting): picked mon.noname-c con 0x7fdb0c0af8e0 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 10 monclient(hunting): picked mon.noname-a con 0x7fdb0c0b0860 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 10 monclient(hunting): picked mon.noname-b con 0x7fdb0c13ef30 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0c135350 con 0x7fdb0c0b0860 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0c14a6a0 con 0x7fdb0c13ef30 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0c06dc90 con 0x7fdb0c0af8e0 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.143+0000 7fdb13286640 10 monclient(hunting): authenticate will time out at 1493.340840s 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.144+0000 7fdb10821640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdb0c0b0860 0x7fdb0c1369c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51092/0 (socket says 172.21.15.98:51092) 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.144+0000 7fdb10821640 1 -- 172.21.15.98:0/1494137152 learned_addr learned my addr 172.21.15.98:0/1494137152 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.144+0000 7fdb01ffb640 10 client.?.objecter ms_handle_connect 0x7fdb0c0b0860 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.144+0000 7fdb01ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:33.136 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277032425 0 0) 0x7fdb0c135350 con 0x7fdb0c0b0860 2024-03-29T10:54:33.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:33.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:33.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:33.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 10 monclient(hunting): my global_id is 4569 2024-03-29T10:54:33.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaf0004eb0 con 0x7fdb0c0b0860 2024-03-29T10:54:33.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 893945357 0 0) 0x7fdaf0004eb0 con 0x7fdb0c0b0860 2024-03-29T10:54:33.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb0c135350 con 0x7fdb0c0b0860 2024-03-29T10:54:33.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fdafc0017d0 con 0x7fdb0c0b0860 2024-03-29T10:54:33.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:33.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:33.137 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:33.137 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:33.137 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:33.138 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:33.138 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:33.138 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:33.138 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:33.138 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:33.138 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:33.138 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:33.138 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:33.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:33.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.145+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 85438072 0 0) 0x7fdb0c135350 con 0x7fdb0c0b0860 2024-03-29T10:54:33.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.146+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 >> v1:172.21.15.98:6790/0 conn(0x7fdb0c0af8e0 legacy=0x7fdb0c148fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:33.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.146+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 >> v1:172.21.15.189:6789/0 conn(0x7fdb0c13ef30 legacy=0x7fdb0c1370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:33.138 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.146+0000 7fdb01ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.146+0000 7fdb01ffb640 1 monclient: found mon.a 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.146+0000 7fdb01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.146+0000 7fdb01ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.146+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c1377e0 con 0x7fdb0c0b0860 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb01ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb01ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb01ffb640 10 client.?.objecter ms_handle_connect 0x7fdb0c13ef30 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb01ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb13286640 5 monclient: authenticate success, global_id 4569 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb13286640 10 monclient: _renew_subs 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb13286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb13286640 1 -- 172.21.15.98:0/1494137152 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0c071990 con 0x7fdb0c0b0860 2024-03-29T10:54:33.139 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb13286640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb13286640 10 monclient: _renew_subs 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb13286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb13286640 1 -- 172.21.15.98:0/1494137152 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0c1496b0 con 0x7fdb0c0b0860 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.147+0000 7fdb13286640 1 librados: init done 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.148+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdafc001930 con 0x7fdb0c0b0860 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.148+0000 7fdb01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.148+0000 7fdb037fe640 4 set_mon_vals no callback set 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.149+0000 7fdb13286640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.149+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fdafc004a40 con 0x7fdb0c0b0860 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.149+0000 7fdb13286640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.149+0000 7fdb13286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.149+0000 7fdb13286640 1 -- 172.21.15.98:0/1494137152 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaec005080 con 0x7fdb0c0b0860 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.149+0000 7fdb01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:33.140 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.149+0000 7fdb01ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.149+0000 7fdb01ffb640 10 monclient: dump: 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.149+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fdafc004c60 con 0x7fdb0c0b0860 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.151+0000 7fdb11022640 10 monclient: get_auth_request con 0x7fdaf006dc50 auth_method 0 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.151+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 646403362 0 0) 0x7fdafc084aa0 con 0x7fdb0c0b0860 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.151+0000 7fdb01ffb640 10 client.4569.objecter ms_dispatch 0x7fdb0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:33.141 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.151+0000 7fdb01ffb640 3 client.4569.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:33.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.151+0000 7fdb01ffb640 3 client.4569.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:33.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.151+0000 7fdb01ffb640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-29T10:54:33.142 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.152+0000 7fdb01ffb640 10 client.4569.objecter ms_handle_connect 0x7fdaf006dc50 2024-03-29T10:54:33.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.158+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdafc050460 con 0x7fdb0c0b0860 2024-03-29T10:54:33.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.158+0000 7fdb01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:33.143 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.158+0000 7fdb01ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:33.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.478+0000 7fdb13286640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:33.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.478+0000 7fdb13286640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:33.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.478+0000 7fdb13286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:33.460 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.478+0000 7fdb13286640 1 -- 172.21.15.98:0/1494137152 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdaec005330 con 0x7fdb0c0b0860 2024-03-29T10:54:33.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.482+0000 7fdb01ffb640 1 -- 172.21.15.98:0/1494137152 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1051361656) 0x7fdafc052390 con 0x7fdb0c0b0860 2024-03-29T10:54:33.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.482+0000 7fdb01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:33.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.482+0000 7fdb01ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:33.463 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:54:33.463 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:44:36.286354+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:36.286354+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:33.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.485+0000 7fdb13286640 10 librados: watch_flush enter 2024-03-29T10:54:33.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.485+0000 7fdb13286640 10 librados: watch_flush exit 2024-03-29T10:54:33.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.485+0000 7fdb13286640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-29T10:54:33.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.485+0000 7fdb13286640 10 client.4569.objecter successfully canceled tick 2024-03-29T10:54:33.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.486+0000 7fdb13286640 1 -- 172.21.15.98:0/1494137152 >> v1:172.21.15.98:6816/1481799426 conn(0x7fdaf006dc50 legacy=0x7fdaf0070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:33.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.486+0000 7fdb13286640 10 monclient: shutdown 2024-03-29T10:54:33.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.486+0000 7fdb13286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:33.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.486+0000 7fdb13286640 1 -- 172.21.15.98:0/1494137152 >> v1:172.21.15.98:6789/0 conn(0x7fdb0c0b0860 legacy=0x7fdb0c1369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:33.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.486+0000 7fdb13286640 1 -- 172.21.15.98:0/1494137152 shutdown_connections 2024-03-29T10:54:33.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.486+0000 7fdb13286640 1 -- 172.21.15.98:0/1494137152 >> 172.21.15.98:0/1494137152 conn(0x7fdb0c06b660 msgr2=0x7fdb0c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:33.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.486+0000 7fdb13286640 1 -- 172.21.15.98:0/1494137152 shutdown_connections 2024-03-29T10:54:33.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.486+0000 7fdb13286640 1 -- 172.21.15.98:0/1494137152 wait complete. 2024-03-29T10:54:33.471 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:33.488+0000 7fdb13286640 1 librados: shutdown 2024-03-29T10:54:33.485 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:40.292 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:40.438 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.457+0000 7fba965e8640 10 monclient: get_monmap_and_config 2024-03-29T10:54:40.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.457+0000 7fba965e8640 10 monclient: build_initial_monmap 2024-03-29T10:54:40.439 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.457+0000 7fba965e8640 10 monclient: monmap: 2024-03-29T10:54:40.439 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:40.439 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:40.439 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:54:40.458410+0000 2024-03-29T10:54:40.439 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:54:40.458410+0000 2024-03-29T10:54:40.439 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:40.439 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:40.439 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:40.439 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:54:40.439 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:54:40.439 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:40.440 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.458+0000 7fba965e8640 1 Processor -- start 2024-03-29T10:54:40.440 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.459+0000 7fba965e8640 1 -- start start 2024-03-29T10:54:40.440 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.459+0000 7fba965e8640 10 monclient: init 2024-03-29T10:54:40.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.460+0000 7fba965e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:40.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.460+0000 7fba965e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:40.442 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.460+0000 7fba965e8640 10 monclient(hunting): picked mon.noname-a con 0x7fba88006890 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:40.443 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.460+0000 7fba965e8640 10 monclient(hunting): picked mon.noname-b con 0x7fba88006c70 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:40.443 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.461+0000 7fba955e6640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fba88006890 0x7fba880c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49420/0 (socket says 172.21.15.98:49420) 2024-03-29T10:54:40.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.462+0000 7fba955e6640 1 -- 172.21.15.98:0/713266471 learned_addr learned my addr 172.21.15.98:0/713266471 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:40.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.462+0000 7fba965e8640 10 monclient(hunting): picked mon.noname-c con 0x7fba8802b4c0 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:40.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.462+0000 7fba965e8640 1 -- 172.21.15.98:0/713266471 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8802c830 con 0x7fba88006890 2024-03-29T10:54:40.444 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.462+0000 7fba965e8640 1 -- 172.21.15.98:0/713266471 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8802afa0 con 0x7fba8802b4c0 2024-03-29T10:54:40.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.462+0000 7fba965e8640 1 -- 172.21.15.98:0/713266471 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8802aba0 con 0x7fba88006c70 2024-03-29T10:54:40.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.462+0000 7fba965e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:40.445 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.462+0000 7fba965e8640 10 monclient(hunting): authenticate will time out at 1500.659566s 2024-03-29T10:54:40.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 1 -- 172.21.15.98:0/713266471 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004339999 0 0) 0x7fba8802c830 con 0x7fba88006890 2024-03-29T10:54:40.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:40.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:40.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:40.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 10 monclient(hunting): my global_id is 4572 2024-03-29T10:54:40.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 1 -- 172.21.15.98:0/713266471 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c003520 con 0x7fba88006890 2024-03-29T10:54:40.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 1 -- 172.21.15.98:0/713266471 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240686965 0 0) 0x7fba8802afa0 con 0x7fba8802b4c0 2024-03-29T10:54:40.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:40.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:40.446 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:40.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 10 monclient(hunting): my global_id is 4582 2024-03-29T10:54:40.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 1 -- 172.21.15.98:0/713266471 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba8802c830 con 0x7fba8802b4c0 2024-03-29T10:54:40.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 1 -- 172.21.15.98:0/713266471 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3659071085 0 0) 0x7fba7c003520 con 0x7fba88006890 2024-03-29T10:54:40.447 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 1 -- 172.21.15.98:0/713266471 >> v1:172.21.15.98:6790/0 conn(0x7fba88006c70 legacy=0x7fba880c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:40.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 1 -- 172.21.15.98:0/713266471 >> v1:172.21.15.189:6789/0 conn(0x7fba8802b4c0 legacy=0x7fba880cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:40.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 1 monclient: found mon.noname-a 2024-03-29T10:54:40.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:40.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:54:40.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.465+0000 7fba87fff640 1 -- 172.21.15.98:0/713266471 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8802ad30 con 0x7fba88006890 2024-03-29T10:54:40.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba87fff640 10 monclient: _finish_auth 0 2024-03-29T10:54:40.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba87fff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:40.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:40.448 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba87fff640 1 -- 172.21.15.98:0/713266471 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fba8c001690 con 0x7fba88006890 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba87fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba87fff640 10 monclient: dump: 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:40.449 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba87fff640 1 -- 172.21.15.98:0/713266471 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba8c001930 con 0x7fba88006890 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba87fff640 1 -- 172.21.15.98:0/713266471 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fba8c001d00 con 0x7fba88006890 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba87fff640 10 monclient: dump: 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:40.450 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:40.451 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:40.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba965e8640 5 monclient: authenticate success, global_id 4572 2024-03-29T10:54:40.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba965e8640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:40.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.466+0000 7fba965e8640 4 set_mon_vals no callback set 2024-03-29T10:54:40.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.467+0000 7fba965e8640 10 monclient: shutdown 2024-03-29T10:54:40.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.467+0000 7fba965e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:40.451 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.467+0000 7fba965e8640 1 -- 172.21.15.98:0/713266471 >> v1:172.21.15.98:6789/0 conn(0x7fba88006890 legacy=0x7fba880c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.469+0000 7fba965e8640 1 -- 172.21.15.98:0/713266471 shutdown_connections 2024-03-29T10:54:40.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.469+0000 7fba965e8640 1 -- 172.21.15.98:0/713266471 >> 172.21.15.98:0/713266471 conn(0x7fba880058f0 msgr2=0x7fba88005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:40.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.470+0000 7fba965e8640 1 -- 172.21.15.98:0/713266471 shutdown_connections 2024-03-29T10:54:40.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.470+0000 7fba965e8640 1 -- 172.21.15.98:0/713266471 wait complete. 2024-03-29T10:54:40.452 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.471+0000 7fba965e8640 10 monclient: build_initial_monmap 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.471+0000 7fba965e8640 10 monclient: monmap: 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.471+0000 7fba965e8640 1 librados: starting msgr at 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.471+0000 7fba965e8640 1 librados: starting objecter 2024-03-29T10:54:40.453 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.472+0000 7fba965e8640 1 Processor -- start 2024-03-29T10:54:40.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.473+0000 7fba965e8640 1 -- start start 2024-03-29T10:54:40.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.473+0000 7fba965e8640 1 librados: setting wanted keys 2024-03-29T10:54:40.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.473+0000 7fba965e8640 1 librados: calling monclient init 2024-03-29T10:54:40.454 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.473+0000 7fba965e8640 10 monclient: init 2024-03-29T10:54:40.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.474+0000 7fba965e8640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:40.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.474+0000 7fba965e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:40.455 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.474+0000 7fba965e8640 10 monclient(hunting): picked mon.noname-b con 0x7fba88006890 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:40.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.474+0000 7fba965e8640 10 monclient(hunting): picked mon.noname-c con 0x7fba8802b4c0 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:40.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.477+0000 7fba965e8640 10 monclient(hunting): picked mon.noname-a con 0x7fba88030da0 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:40.462 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.477+0000 7fba95de7640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fba88030da0 0x7fba881589c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52666/0 (socket says 172.21.15.98:52666) 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.477+0000 7fba95de7640 1 -- 172.21.15.98:0/733649291 learned_addr learned my addr 172.21.15.98:0/733649291 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.478+0000 7fba965e8640 1 -- 172.21.15.98:0/733649291 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8802d830 con 0x7fba88030da0 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.478+0000 7fba965e8640 1 -- 172.21.15.98:0/733649291 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba880be6c0 con 0x7fba88006890 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.478+0000 7fba965e8640 1 -- 172.21.15.98:0/733649291 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba880be850 con 0x7fba8802b4c0 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.478+0000 7fba965e8640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.478+0000 7fba965e8640 10 monclient(hunting): authenticate will time out at 1500.675373s 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.478+0000 7fba85ffb640 10 client.?.objecter ms_handle_connect 0x7fba88030da0 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.478+0000 7fba85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.478+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37203833 0 0) 0x7fba8802d830 con 0x7fba88030da0 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.478+0000 7fba85ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.478+0000 7fba85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.478+0000 7fba85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.478+0000 7fba85ffb640 10 monclient(hunting): my global_id is 4575 2024-03-29T10:54:40.463 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba74004eb0 con 0x7fba88030da0 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 10 client.?.objecter ms_handle_connect 0x7fba8802b4c0 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444771954 0 0) 0x7fba880be850 con 0x7fba8802b4c0 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 10 monclient(hunting): my global_id is 4634 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba8802d830 con 0x7fba8802b4c0 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3360530511 0 0) 0x7fba74004eb0 con 0x7fba88030da0 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba880be850 con 0x7fba88030da0 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fba9004fdd0 con 0x7fba88030da0 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:40.464 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.479+0000 7fba85ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.480+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 <== mon.0 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2321865928 0 0) 0x7fba880be850 con 0x7fba88030da0 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.480+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 >> v1:172.21.15.98:6790/0 conn(0x7fba8802b4c0 legacy=0x7fba880bdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.480+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 >> v1:172.21.15.189:6789/0 conn(0x7fba88006890 legacy=0x7fba88030690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.480+0000 7fba85ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:40.465 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.481+0000 7fba85ffb640 1 monclient: found mon.a 2024-03-29T10:54:40.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.481+0000 7fba85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:40.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.481+0000 7fba85ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:40.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.481+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba880be9e0 con 0x7fba88030da0 2024-03-29T10:54:40.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.481+0000 7fba85ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:40.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.481+0000 7fba85ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:40.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.481+0000 7fba85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:40.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.481+0000 7fba85ffb640 10 client.?.objecter ms_handle_connect 0x7fba88006890 2024-03-29T10:54:40.466 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.481+0000 7fba85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:40.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.482+0000 7fba965e8640 5 monclient: authenticate success, global_id 4575 2024-03-29T10:54:40.467 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.482+0000 7fba965e8640 10 monclient: _renew_subs 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.482+0000 7fba965e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.482+0000 7fba965e8640 1 -- 172.21.15.98:0/733649291 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba88031460 con 0x7fba88030da0 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.483+0000 7fba965e8640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.483+0000 7fba965e8640 10 monclient: _renew_subs 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.483+0000 7fba965e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.483+0000 7fba965e8640 1 -- 172.21.15.98:0/733649291 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba881590d0 con 0x7fba88030da0 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.483+0000 7fba965e8640 1 librados: init done 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.483+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 <== mon.0 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba9004f400 con 0x7fba88030da0 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.483+0000 7fba85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.483+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 <== mon.0 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fba90075f10 con 0x7fba88030da0 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.483+0000 7fba85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.483+0000 7fba85ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.483+0000 7fba85ffb640 10 monclient: dump: 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:40.468 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.483+0000 7fba877fe640 4 set_mon_vals no callback set 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.484+0000 7fba7b7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.484+0000 7fba7b7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.484+0000 7fba7b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.484+0000 7fba7b7fe640 1 -- 172.21.15.98:0/733649291 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8802a520 con 0x7fba88030da0 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.485+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 <== mon.0 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fba90067bf0 con 0x7fba88030da0 2024-03-29T10:54:40.469 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.486+0000 7fba955e6640 10 monclient: get_auth_request con 0x7fba7406da70 auth_method 0 2024-03-29T10:54:40.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.486+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 <== mon.0 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2292924465 0 0) 0x7fba900ff550 con 0x7fba88030da0 2024-03-29T10:54:40.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.486+0000 7fba85ffb640 10 client.4575.objecter ms_dispatch 0x7fba90000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:54:40.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.486+0000 7fba85ffb640 3 client.4575.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:54:40.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.486+0000 7fba85ffb640 3 client.4575.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:54:40.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.486+0000 7fba85ffb640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-29T10:54:40.470 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.486+0000 7fba85ffb640 10 client.4575.objecter ms_handle_connect 0x7fba7406da70 2024-03-29T10:54:40.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.495+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 <== mon.0 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba900cae30 con 0x7fba88030da0 2024-03-29T10:54:40.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.495+0000 7fba85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:40.476 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.495+0000 7fba85ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:40.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.823+0000 7fba7b7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:40.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.823+0000 7fba7b7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:40.804 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.823+0000 7fba7b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.98:6789/0 2024-03-29T10:54:40.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.823+0000 7fba7b7fe640 1 -- 172.21.15.98:0/733649291 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba88004cd0 con 0x7fba88030da0 2024-03-29T10:54:40.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.824+0000 7fba85ffb640 1 -- 172.21.15.98:0/733649291 <== mon.0 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2920969763) 0x7fba900ccd60 con 0x7fba88030da0 2024-03-29T10:54:40.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.824+0000 7fba85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:40.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.824+0000 7fba85ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:40.807 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:54:40.807 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:54:39.048768+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:39.048768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:40.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.829+0000 7fba7b7fe640 10 librados: watch_flush enter 2024-03-29T10:54:40.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.829+0000 7fba7b7fe640 10 librados: watch_flush exit 2024-03-29T10:54:40.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.829+0000 7fba7b7fe640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-29T10:54:40.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.829+0000 7fba7b7fe640 10 client.4575.objecter successfully canceled tick 2024-03-29T10:54:40.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.830+0000 7fba7b7fe640 1 -- 172.21.15.98:0/733649291 >> v1:172.21.15.98:6816/1481799426 conn(0x7fba7406da70 legacy=0x7fba7406ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.830+0000 7fba7b7fe640 10 monclient: shutdown 2024-03-29T10:54:40.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.830+0000 7fba7b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:40.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.830+0000 7fba7b7fe640 1 -- 172.21.15.98:0/733649291 >> v1:172.21.15.98:6789/0 conn(0x7fba88030da0 legacy=0x7fba881589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.830+0000 7fba7b7fe640 1 -- 172.21.15.98:0/733649291 shutdown_connections 2024-03-29T10:54:40.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.830+0000 7fba7b7fe640 1 -- 172.21.15.98:0/733649291 >> 172.21.15.98:0/733649291 conn(0x7fba880058f0 msgr2=0x7fba88038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:40.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.831+0000 7fba7b7fe640 1 -- 172.21.15.98:0/733649291 shutdown_connections 2024-03-29T10:54:40.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.831+0000 7fba7b7fe640 1 -- 172.21.15.98:0/733649291 wait complete. 2024-03-29T10:54:40.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:40.832+0000 7fba7b7fe640 1 librados: shutdown 2024-03-29T10:54:40.830 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:47.638 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:47.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.810+0000 7f562a315640 10 monclient: get_monmap_and_config 2024-03-29T10:54:47.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.810+0000 7f562a315640 10 monclient: build_initial_monmap 2024-03-29T10:54:47.792 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.810+0000 7f562a315640 10 monclient: monmap: 2024-03-29T10:54:47.793 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:47.793 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:47.793 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:54:47.811562+0000 2024-03-29T10:54:47.793 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:54:47.811562+0000 2024-03-29T10:54:47.793 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:47.793 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:47.793 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:47.793 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:54:47.793 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:54:47.793 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:47.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.813+0000 7f562a315640 1 Processor -- start 2024-03-29T10:54:47.795 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.813+0000 7f562a315640 1 -- start start 2024-03-29T10:54:47.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.813+0000 7f562a315640 10 monclient: init 2024-03-29T10:54:47.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.814+0000 7f562a315640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:47.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.814+0000 7f562a315640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:47.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.814+0000 7f562a315640 10 monclient(hunting): picked mon.noname-c con 0x7f56240a9930 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:47.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.814+0000 7f562a315640 10 monclient(hunting): picked mon.noname-a con 0x7f5624149dc0 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:47.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.814+0000 7f562a315640 10 monclient(hunting): picked mon.noname-b con 0x7f562414ff40 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:47.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.814+0000 7f562a315640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56240aae60 con 0x7f5624149dc0 2024-03-29T10:54:47.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.814+0000 7f562a315640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56240b4310 con 0x7f56240a9930 2024-03-29T10:54:47.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.814+0000 7f562a315640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56240afdd0 con 0x7f562414ff40 2024-03-29T10:54:47.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.814+0000 7f562a315640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:47.796 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.814+0000 7f562a315640 10 monclient(hunting): authenticate will time out at 1508.011252s 2024-03-29T10:54:47.798 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.817+0000 7f56237fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5624149dc0 0x7f562414a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41955/0 (socket says 172.21.15.98:41955) 2024-03-29T10:54:47.799 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.817+0000 7f56237fe640 1 -- 172.21.15.98:0/1443272686 learned_addr learned my addr 172.21.15.98:0/1443272686 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:47.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140760227 0 0) 0x7f56240aae60 con 0x7f5624149dc0 2024-03-29T10:54:47.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:47.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 10 monclient(hunting): my global_id is 4578 2024-03-29T10:54:47.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c003520 con 0x7f5624149dc0 2024-03-29T10:54:47.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349345189 0 0) 0x7f56240afdd0 con 0x7f562414ff40 2024-03-29T10:54:47.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.805 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:47.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 10 monclient(hunting): my global_id is 4637 2024-03-29T10:54:47.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56240aae60 con 0x7f562414ff40 2024-03-29T10:54:47.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355751589 0 0) 0x7f56240b4310 con 0x7f56240a9930 2024-03-29T10:54:47.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:47.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 10 monclient(hunting): my global_id is 4588 2024-03-29T10:54:47.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.820+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56240afdd0 con 0x7f56240a9930 2024-03-29T10:54:47.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1296778397 0 0) 0x7f560c003520 con 0x7f5624149dc0 2024-03-29T10:54:47.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 >> v1:172.21.15.98:6790/0 conn(0x7f562414ff40 legacy=0x7f5624152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 >> v1:172.21.15.189:6789/0 conn(0x7f56240a9930 legacy=0x7f5624143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f5622ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:47.806 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f5622ffd640 1 monclient: found mon.noname-a 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f5622ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f5622ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562414a950 con 0x7f5624149dc0 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f5622ffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f5622ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f5622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f56237fe640 1 -- 172.21.15.98:0/1443272686 >> v1:172.21.15.98:6789/0 conn(0x7f5624149dc0 legacy=0x7f562414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f56237fe640 1 -- 172.21.15.98:0/1443272686 >> v1:172.21.15.98:6789/0 conn(0x7f5624149dc0 legacy=0x7f562414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f56237fe640 1 --1- 172.21.15.98:0/1443272686 >> v1:172.21.15.98:6789/0 conn(0x7f5624149dc0 0x7f562414a240 :-1 s=OPENED pgs=1041140 cs=1 l=1).handle_message read tag failed 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.821+0000 7f56237fe640 1 --1- 172.21.15.98:0/1443272686 >> v1:172.21.15.98:6789/0 conn(0x7f5624149dc0 0x7f562414a240 :-1 s=OPENED pgs=1041140 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f562a315640 5 monclient: authenticate success, global_id 4578 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f562a315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f5622ffd640 10 monclient: ms_handle_reset current mon v1:172.21.15.98:6789/0 2024-03-29T10:54:47.807 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f5622ffd640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:47.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 >> v1:172.21.15.98:6789/0 conn(0x7f5624149dc0 legacy=0x7f562414a240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:54:47.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f5622ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:47.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f5622ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f56240a9930 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:47.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f5622ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f560c0079e0 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:47.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f5622ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f562414ff40 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:47.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c003520 con 0x7f560c0079e0 2024-03-29T10:54:47.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56240b4310 con 0x7f56240a9930 2024-03-29T10:54:47.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560c010fb0 con 0x7f562414ff40 2024-03-29T10:54:47.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f5622ffd640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:47.808 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.822+0000 7f5622ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.98:6789/0 2024-03-29T10:54:47.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246772267 0 0) 0x7f560c003520 con 0x7f560c0079e0 2024-03-29T10:54:47.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T10:54:47.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 --> v1:172.21.15.98:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f560c00eeb0 con 0x7f560c0079e0 2024-03-29T10:54:47.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 531716169 0 0) 0x7f560c00eeb0 con 0x7f560c0079e0 2024-03-29T10:54:47.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 >> v1:172.21.15.98:6790/0 conn(0x7f562414ff40 legacy=0x7f560c00d5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:47.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 >> v1:172.21.15.189:6789/0 conn(0x7f56240a9930 legacy=0x7f560c004ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.811 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 1 monclient: found mon.noname-a 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c0122d0 con 0x7f560c0079e0 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5618001690 con 0x7f560c0079e0 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f5622ffd640 10 monclient: dump: 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:47.812 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:47.813 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:47.813 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:47.813 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:47.813 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:47.813 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:47.813 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:47.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.829+0000 7f562a315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:47.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.830+0000 7f5622ffd640 1 -- 172.21.15.98:0/1443272686 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56180018b0 con 0x7f560c0079e0 2024-03-29T10:54:47.813 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.830+0000 7f5622ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:47.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.830+0000 7f562a315640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:47.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.830+0000 7f562a315640 4 set_mon_vals no callback set 2024-03-29T10:54:47.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.831+0000 7f562a315640 10 monclient: shutdown 2024-03-29T10:54:47.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.831+0000 7f562a315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:47.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.831+0000 7f562a315640 1 -- 172.21.15.98:0/1443272686 >> v1:172.21.15.98:6789/0 conn(0x7f560c0079e0 legacy=0x7f560c009dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.814 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.833+0000 7f562a315640 1 -- 172.21.15.98:0/1443272686 shutdown_connections 2024-03-29T10:54:47.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.833+0000 7f562a315640 1 -- 172.21.15.98:0/1443272686 >> 172.21.15.98:0/1443272686 conn(0x7f56240b35e0 msgr2=0x7f56240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:47.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.833+0000 7f562a315640 1 -- 172.21.15.98:0/1443272686 shutdown_connections 2024-03-29T10:54:47.815 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.833+0000 7f562a315640 1 -- 172.21.15.98:0/1443272686 wait complete. 2024-03-29T10:54:47.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.834+0000 7f562a315640 10 monclient: build_initial_monmap 2024-03-29T10:54:47.816 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.834+0000 7f562a315640 10 monclient: monmap: 2024-03-29T10:54:47.816 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:47.816 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:47.816 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:54:47.816 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:54:47.816 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:47.817 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:47.817 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:47.817 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:54:47.817 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:54:47.817 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:47.817 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.835+0000 7f562a315640 1 librados: starting msgr at 2024-03-29T10:54:47.817 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.835+0000 7f562a315640 1 librados: starting objecter 2024-03-29T10:54:47.817 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.835+0000 7f562a315640 1 Processor -- start 2024-03-29T10:54:47.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.835+0000 7f562a315640 1 -- start start 2024-03-29T10:54:47.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.835+0000 7f562a315640 1 librados: setting wanted keys 2024-03-29T10:54:47.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.835+0000 7f562a315640 1 librados: calling monclient init 2024-03-29T10:54:47.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.835+0000 7f562a315640 10 monclient: init 2024-03-29T10:54:47.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.836+0000 7f562a315640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:47.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.836+0000 7f562a315640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:47.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.836+0000 7f562a315640 10 monclient(hunting): picked mon.noname-b con 0x7f560c0079e0 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:47.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.836+0000 7f562a315640 10 monclient(hunting): picked mon.noname-c con 0x7f56240a9930 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:47.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.836+0000 7f562a315640 10 monclient(hunting): picked mon.noname-a con 0x7f562414ff40 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:47.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.836+0000 7f562a315640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562414aae0 con 0x7f562414ff40 2024-03-29T10:54:47.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.836+0000 7f562a315640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5624149330 con 0x7f560c0079e0 2024-03-29T10:54:47.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.836+0000 7f562a315640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562413bd30 con 0x7f56240a9930 2024-03-29T10:54:47.822 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.836+0000 7f562a315640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:47.823 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.836+0000 7f562a315640 10 monclient(hunting): authenticate will time out at 1508.033698s 2024-03-29T10:54:47.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.842+0000 7f5623fff640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f560c0079e0 0x7f56241de700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.98:59025/0 (socket says 172.21.15.98:59025) 2024-03-29T10:54:47.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.842+0000 7f5623fff640 1 -- 172.21.15.98:0/878557860 learned_addr learned my addr 172.21.15.98:0/878557860 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:47.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 10 client.?.objecter ms_handle_connect 0x7f560c0079e0 2024-03-29T10:54:47.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:47.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739897084 0 0) 0x7f5624149330 con 0x7f560c0079e0 2024-03-29T10:54:47.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:47.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 10 monclient(hunting): my global_id is 4591 2024-03-29T10:54:47.826 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5610004eb0 con 0x7f560c0079e0 2024-03-29T10:54:47.827 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 10 client.?.objecter ms_handle_connect 0x7f56240a9930 2024-03-29T10:54:47.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:47.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260037526 0 0) 0x7f562413bd30 con 0x7f56240a9930 2024-03-29T10:54:47.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:47.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.845+0000 7f5620ff9640 10 monclient(hunting): my global_id is 4640 2024-03-29T10:54:47.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.846+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5624149330 con 0x7f56240a9930 2024-03-29T10:54:47.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.846+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3642709645 0 0) 0x7f5624149330 con 0x7f56240a9930 2024-03-29T10:54:47.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.846+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f562413bd30 con 0x7f56240a9930 2024-03-29T10:54:47.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.846+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5618001020 con 0x7f56240a9930 2024-03-29T10:54:47.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.846+0000 7f5620ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:47.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.846+0000 7f5620ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:47.828 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.846+0000 7f5620ff9640 10 monclient(hunting): dump: 2024-03-29T10:54:47.829 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:47.829 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:47.829 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:47.829 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:47.829 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:47.829 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:47.829 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:47.829 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:47.829 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:47.829 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:47.829 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.846+0000 7f5620ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:47.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 204176863 0 0) 0x7f5610004eb0 con 0x7f560c0079e0 2024-03-29T10:54:47.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5624149330 con 0x7f560c0079e0 2024-03-29T10:54:47.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f561c001cb0 con 0x7f560c0079e0 2024-03-29T10:54:47.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:47.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:47.830 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 10 monclient(hunting): dump: 2024-03-29T10:54:47.830 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:47.830 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1994049424 0 0) 0x7f562413bd30 con 0x7f56240a9930 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 >> v1:172.21.15.189:6789/0 conn(0x7f560c0079e0 legacy=0x7f56241de700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 >> v1:172.21.15.98:6789/0 conn(0x7f562414ff40 legacy=0x7f562413b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 1 monclient: found mon.c 2024-03-29T10:54:47.831 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:47.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:47.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56241474d0 con 0x7f56240a9930 2024-03-29T10:54:47.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 10 monclient: _finish_auth 0 2024-03-29T10:54:47.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:54:47.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f5620ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:47.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f562a315640 5 monclient: authenticate success, global_id 4640 2024-03-29T10:54:47.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f562a315640 10 monclient: _renew_subs 2024-03-29T10:54:47.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f562a315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:47.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f562a315640 1 -- 172.21.15.98:0/878557860 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5624148890 con 0x7f56240a9930 2024-03-29T10:54:47.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f562a315640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:47.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f562a315640 10 monclient: _renew_subs 2024-03-29T10:54:47.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f562a315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:47.832 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f562a315640 1 -- 172.21.15.98:0/878557860 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56241dee10 con 0x7f56240a9930 2024-03-29T10:54:47.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.847+0000 7f562a315640 1 librados: init done 2024-03-29T10:54:47.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.849+0000 7f562a315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:47.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.849+0000 7f562a315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:47.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.849+0000 7f562a315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:47.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.849+0000 7f562a315640 1 -- 172.21.15.98:0/878557860 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56241df0c0 con 0x7f56240a9930 2024-03-29T10:54:47.833 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.849+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5618001cb0 con 0x7f56240a9930 2024-03-29T10:54:47.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.849+0000 7f5620ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:47.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.849+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5618004b00 con 0x7f56240a9930 2024-03-29T10:54:47.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.849+0000 7f5620ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:47.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.849+0000 7f5620ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:47.834 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.849+0000 7f5620ff9640 10 monclient: dump: 2024-03-29T10:54:47.834 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:47.834 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:47.834 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:47.834 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:47.834 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:47.834 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:47.834 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:47.834 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:47.835 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:47.835 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:47.835 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.849+0000 7f56227fc640 4 set_mon_vals no callback set 2024-03-29T10:54:47.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.852+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f561801ae40 con 0x7f56240a9930 2024-03-29T10:54:47.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.853+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2292924465 0 0) 0x7f5618084b90 con 0x7f56240a9930 2024-03-29T10:54:47.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.853+0000 7f5620ff9640 10 client.4640.objecter ms_dispatch 0x7f5624000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:54:47.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.853+0000 7f5620ff9640 3 client.4640.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:54:47.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.853+0000 7f5620ff9640 3 client.4640.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:54:47.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.853+0000 7f5620ff9640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-29T10:54:47.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.853+0000 7f5623fff640 10 monclient: get_auth_request con 0x7f561006df20 auth_method 0 2024-03-29T10:54:47.836 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.853+0000 7f5620ff9640 10 client.4640.objecter ms_handle_connect 0x7f561006df20 2024-03-29T10:54:47.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.858+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5618050470 con 0x7f56240a9930 2024-03-29T10:54:47.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.858+0000 7f5620ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:47.839 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:47.858+0000 7f5620ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:48.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.177+0000 7f562a315640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:48.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.177+0000 7f562a315640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:48.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.177+0000 7f562a315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:48.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.177+0000 7f562a315640 1 -- 172.21.15.98:0/878557860 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5624075f10 con 0x7f56240a9930 2024-03-29T10:54:48.159 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:54:48.159 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:54:39.048768+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:39.048768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:48.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.178+0000 7f5620ff9640 1 -- 172.21.15.98:0/878557860 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2920969763) 0x7f56180523a0 con 0x7f56240a9930 2024-03-29T10:54:48.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.178+0000 7f5620ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:48.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.178+0000 7f5620ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:48.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.181+0000 7f562a315640 10 librados: watch_flush enter 2024-03-29T10:54:48.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.181+0000 7f562a315640 10 librados: watch_flush exit 2024-03-29T10:54:48.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.181+0000 7f562a315640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-29T10:54:48.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.181+0000 7f562a315640 10 client.4640.objecter successfully canceled tick 2024-03-29T10:54:48.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.181+0000 7f562a315640 1 -- 172.21.15.98:0/878557860 >> v1:172.21.15.98:6816/1481799426 conn(0x7f561006df20 legacy=0x7f56100703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:48.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.181+0000 7f562a315640 10 monclient: shutdown 2024-03-29T10:54:48.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.181+0000 7f562a315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:48.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.181+0000 7f562a315640 1 -- 172.21.15.98:0/878557860 >> v1:172.21.15.98:6790/0 conn(0x7f56240a9930 legacy=0x7f562413af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:48.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.183+0000 7f562a315640 1 -- 172.21.15.98:0/878557860 shutdown_connections 2024-03-29T10:54:48.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.183+0000 7f562a315640 1 -- 172.21.15.98:0/878557860 >> 172.21.15.98:0/878557860 conn(0x7f56240b35e0 msgr2=0x7f56240b24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:48.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.184+0000 7f562a315640 1 -- 172.21.15.98:0/878557860 shutdown_connections 2024-03-29T10:54:48.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.185+0000 7f562a315640 1 -- 172.21.15.98:0/878557860 wait complete. 2024-03-29T10:54:48.165 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:48.185+0000 7f562a315640 1 librados: shutdown 2024-03-29T10:54:48.187 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:54.994 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:55.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.177+0000 7f1ad7a3e640 10 monclient: get_monmap_and_config 2024-03-29T10:54:55.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.177+0000 7f1ad7a3e640 10 monclient: build_initial_monmap 2024-03-29T10:54:55.158 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.177+0000 7f1ad7a3e640 10 monclient: monmap: 2024-03-29T10:54:55.158 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:55.158 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:55.158 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:54:55.177973+0000 2024-03-29T10:54:55.158 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:54:55.177973+0000 2024-03-29T10:54:55.158 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:55.159 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:55.159 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:55.159 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:54:55.159 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:54:55.159 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:55.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.178+0000 7f1ad7a3e640 1 Processor -- start 2024-03-29T10:54:55.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.178+0000 7f1ad7a3e640 1 -- start start 2024-03-29T10:54:55.159 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.178+0000 7f1ad7a3e640 10 monclient: init 2024-03-29T10:54:55.160 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.179+0000 7f1ad7a3e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:55.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.179+0000 7f1ad7a3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:55.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.179+0000 7f1ad7a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f1ad00b0770 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:55.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.179+0000 7f1ad7a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f1ad014a700 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:55.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.179+0000 7f1ad7a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f1ad00ac5a0 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:55.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.179+0000 7f1ad7a3e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad00aba70 con 0x7f1ad014a700 2024-03-29T10:54:55.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.179+0000 7f1ad7a3e640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad00aecb0 con 0x7f1ad00b0770 2024-03-29T10:54:55.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.179+0000 7f1ad7a3e640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad00af150 con 0x7f1ad00ac5a0 2024-03-29T10:54:55.161 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.179+0000 7f1ad7a3e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:55.162 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.179+0000 7f1ad7a3e640 10 monclient(hunting): authenticate will time out at 1515.376751s 2024-03-29T10:54:55.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.183+0000 7f1ad5fdb640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f1ad00ac5a0 0x7f1ad0151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:53130/0 (socket says 172.21.15.98:53130) 2024-03-29T10:54:55.164 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.183+0000 7f1ad5fdb640 1 -- 172.21.15.98:0/3268675187 learned_addr learned my addr 172.21.15.98:0/3268675187 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:55.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.184+0000 7f1acffff640 1 -- 172.21.15.98:0/3268675187 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331988422 0 0) 0x7f1ad00af150 con 0x7f1ad00ac5a0 2024-03-29T10:54:55.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.185+0000 7f1acffff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:55.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.185+0000 7f1acffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:55.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.185+0000 7f1acffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:55.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.185+0000 7f1acffff640 10 monclient(hunting): my global_id is 4643 2024-03-29T10:54:55.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.185+0000 7f1acffff640 1 -- 172.21.15.98:0/3268675187 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab8003520 con 0x7f1ad00ac5a0 2024-03-29T10:54:55.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.185+0000 7f1acffff640 1 -- 172.21.15.98:0/3268675187 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2316372857 0 0) 0x7f1ab8003520 con 0x7f1ad00ac5a0 2024-03-29T10:54:55.166 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.185+0000 7f1acffff640 1 -- 172.21.15.98:0/3268675187 >> v1:172.21.15.189:6789/0 conn(0x7f1ad00b0770 legacy=0x7f1ad0149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:54:55.167 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1acffff640 1 -- 172.21.15.98:0/3268675187 >> v1:172.21.15.98:6789/0 conn(0x7f1ad014a700 legacy=0x7f1ad00abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:55.167 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1acffff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:55.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1acffff640 1 monclient: found mon.noname-b 2024-03-29T10:54:55.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1acffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:55.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1acffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:54:55.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1acffff640 1 -- 172.21.15.98:0/3268675187 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad00ac9d0 con 0x7f1ad00ac5a0 2024-03-29T10:54:55.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1acffff640 10 monclient: _finish_auth 0 2024-03-29T10:54:55.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1acffff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:55.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1acffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:55.168 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1acffff640 1 -- 172.21.15.98:0/3268675187 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f1ac8001a10 con 0x7f1ad00ac5a0 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1acffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1acffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1acffff640 10 monclient: dump: 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1ad7a3e640 5 monclient: authenticate success, global_id 4643 2024-03-29T10:54:55.169 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.187+0000 7f1ad7a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:55.170 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.188+0000 7f1acffff640 1 -- 172.21.15.98:0/3268675187 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ac8001be0 con 0x7f1ad00ac5a0 2024-03-29T10:54:55.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.188+0000 7f1acffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:55.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.188+0000 7f1ad7a3e640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:55.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.188+0000 7f1ad7a3e640 4 set_mon_vals no callback set 2024-03-29T10:54:55.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.188+0000 7f1acffff640 1 -- 172.21.15.98:0/3268675187 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f1ac8002030 con 0x7f1ad00ac5a0 2024-03-29T10:54:55.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.188+0000 7f1acffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:55.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.188+0000 7f1acffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:55.171 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.188+0000 7f1acffff640 10 monclient: dump: 2024-03-29T10:54:55.171 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:55.171 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:55.171 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:55.171 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:55.171 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:55.171 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:55.172 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:55.172 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:55.172 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:55.172 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:55.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.189+0000 7f1ad7a3e640 10 monclient: shutdown 2024-03-29T10:54:55.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.189+0000 7f1ad7a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:55.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.189+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/3268675187 >> v1:172.21.15.98:6790/0 conn(0x7f1ad00ac5a0 legacy=0x7f1ad0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:55.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.189+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/3268675187 shutdown_connections 2024-03-29T10:54:55.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.189+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/3268675187 >> 172.21.15.98:0/3268675187 conn(0x7f1ad006b660 msgr2=0x7f1ad006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:55.172 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.190+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/3268675187 shutdown_connections 2024-03-29T10:54:55.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.190+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/3268675187 wait complete. 2024-03-29T10:54:55.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.191+0000 7f1ad7a3e640 10 monclient: build_initial_monmap 2024-03-29T10:54:55.173 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.191+0000 7f1ad7a3e640 10 monclient: monmap: 2024-03-29T10:54:55.173 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:54:55.173 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:55.173 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:54:55.174 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:54:55.174 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:55.174 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:55.174 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:54:55.174 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:54:55.174 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:54:55.174 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:55.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.192+0000 7f1ad7a3e640 1 librados: starting msgr at 2024-03-29T10:54:55.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.192+0000 7f1ad7a3e640 1 librados: starting objecter 2024-03-29T10:54:55.174 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.192+0000 7f1ad7a3e640 1 Processor -- start 2024-03-29T10:54:55.175 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.192+0000 7f1ad7a3e640 1 -- start start 2024-03-29T10:54:55.175 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.192+0000 7f1ad7a3e640 1 librados: setting wanted keys 2024-03-29T10:54:55.175 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.192+0000 7f1ad7a3e640 1 librados: calling monclient init 2024-03-29T10:54:55.175 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.192+0000 7f1ad7a3e640 10 monclient: init 2024-03-29T10:54:55.175 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.194+0000 7f1ad7a3e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:55.175 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.194+0000 7f1ad7a3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:55.177 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.194+0000 7f1ad7a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f1ad00ac5a0 addr v1:172.21.15.98:6789/0 2024-03-29T10:54:55.178 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.197+0000 7f1ad7a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f1ad00b0770 addr v1:172.21.15.98:6790/0 2024-03-29T10:54:55.180 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.198+0000 7f1ad4fd9640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f1ad00b0770 0x7f1ad0146490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:58204/0 (socket says 172.21.15.98:58204) 2024-03-29T10:54:55.182 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.199+0000 7f1ad7a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f1ad014a700 addr v1:172.21.15.189:6789/0 2024-03-29T10:54:55.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.199+0000 7f1ad7a3e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad01df880 con 0x7f1ad00ac5a0 2024-03-29T10:54:55.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.199+0000 7f1ad4fd9640 1 -- 172.21.15.98:0/1052193226 learned_addr learned my addr 172.21.15.98:0/1052193226 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:54:55.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.200+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/1052193226 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad01dfb40 con 0x7f1ad014a700 2024-03-29T10:54:55.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/1052193226 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad012a720 con 0x7f1ad00b0770 2024-03-29T10:54:55.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1acdffb640 10 client.?.objecter ms_handle_connect 0x7f1ad00ac5a0 2024-03-29T10:54:55.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1acdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:55.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1acdffb640 10 client.?.objecter ms_handle_connect 0x7f1ad00b0770 2024-03-29T10:54:55.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1acdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:55.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1ad7a3e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:55.183 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1ad7a3e640 10 monclient(hunting): authenticate will time out at 1515.399345s 2024-03-29T10:54:55.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141108512 0 0) 0x7f1ad01df880 con 0x7f1ad00ac5a0 2024-03-29T10:54:55.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1acdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:55.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1acdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:55.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1acdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:55.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1acdffb640 10 monclient(hunting): my global_id is 4581 2024-03-29T10:54:55.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1abc004eb0 con 0x7f1ad00ac5a0 2024-03-29T10:54:55.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1acdffb640 10 client.?.objecter ms_handle_connect 0x7f1ad014a700 2024-03-29T10:54:55.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.202+0000 7f1acdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:55.187 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392649826 0 0) 0x7f1ad012a720 con 0x7f1ad00b0770 2024-03-29T10:54:55.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:55.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:55.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:55.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 10 monclient(hunting): my global_id is 4646 2024-03-29T10:54:55.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad01df880 con 0x7f1ad00b0770 2024-03-29T10:54:55.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755975027 0 0) 0x7f1ad01dfb40 con 0x7f1ad014a700 2024-03-29T10:54:55.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:55.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:55.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:55.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 10 monclient(hunting): my global_id is 4594 2024-03-29T10:54:55.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad012a720 con 0x7f1ad014a700 2024-03-29T10:54:55.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3409788382 0 0) 0x7f1ad01df880 con 0x7f1ad00b0770 2024-03-29T10:54:55.188 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ad01dfb40 con 0x7f1ad00b0770 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f1ac40018d0 con 0x7f1ad00b0770 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 10 monclient(hunting): dump: 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:55.189 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.203+0000 7f1acdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1863499881 0 0) 0x7f1ad012a720 con 0x7f1ad014a700 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 --> v1:172.21.15.189:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ad01df880 con 0x7f1ad014a700 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f1ac8001d50 con 0x7f1ad014a700 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 10 monclient(hunting): dump: 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:55.190 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1021937910 0 0) 0x7f1ad01dfb40 con 0x7f1ad00b0770 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 >> v1:172.21.15.189:6789/0 conn(0x7f1ad014a700 legacy=0x7f1ad0128330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 >> v1:172.21.15.98:6789/0 conn(0x7f1ad00ac5a0 legacy=0x7f1ad01df170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 1 monclient: found mon.c 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad0147130 con 0x7f1ad00b0770 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.204+0000 7f1acdffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.205+0000 7f1acdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.205+0000 7f1acdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:55.191 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.205+0000 7f1ad7a3e640 5 monclient: authenticate success, global_id 4646 2024-03-29T10:54:55.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.205+0000 7f1ad7a3e640 10 monclient: _renew_subs 2024-03-29T10:54:55.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.205+0000 7f1ad7a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:55.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.205+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/1052193226 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad0147320 con 0x7f1ad00b0770 2024-03-29T10:54:55.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.205+0000 7f1ad7a3e640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:55.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.206+0000 7f1ad7a3e640 10 monclient: _renew_subs 2024-03-29T10:54:55.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.206+0000 7f1ad7a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:55.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.206+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/1052193226 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad0146d80 con 0x7f1ad00b0770 2024-03-29T10:54:55.192 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.206+0000 7f1ad7a3e640 1 librados: init done 2024-03-29T10:54:55.193 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.207+0000 7f1ad7a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:55.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.207+0000 7f1ad7a3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:55.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.207+0000 7f1ad7a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:55.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.207+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/1052193226 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad00b0b50 con 0x7f1ad00b0770 2024-03-29T10:54:55.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.208+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ac4003610 con 0x7f1ad00b0770 2024-03-29T10:54:55.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.208+0000 7f1acdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:55.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.208+0000 7f1aceffd640 4 set_mon_vals no callback set 2024-03-29T10:54:55.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.209+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f1ac40039a0 con 0x7f1ad00b0770 2024-03-29T10:54:55.194 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.209+0000 7f1acdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.209+0000 7f1acdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.209+0000 7f1acdffb640 10 monclient: dump: 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.211+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f1ac4003b00 con 0x7f1ad00b0770 2024-03-29T10:54:55.195 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.212+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2292924465 0 0) 0x7f1ac40848f0 con 0x7f1ad00b0770 2024-03-29T10:54:55.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.212+0000 7f1acdffb640 10 client.4646.objecter ms_dispatch 0x7f1ad0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:54:55.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.212+0000 7f1acdffb640 3 client.4646.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:54:55.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.212+0000 7f1acdffb640 3 client.4646.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:54:55.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.212+0000 7f1acdffb640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-29T10:54:55.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.212+0000 7f1ad57da640 10 monclient: get_auth_request con 0x7f1abc06dec0 auth_method 0 2024-03-29T10:54:55.196 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.213+0000 7f1acdffb640 10 client.4646.objecter ms_handle_connect 0x7f1abc06dec0 2024-03-29T10:54:55.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.218+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ac40501d0 con 0x7f1ad00b0770 2024-03-29T10:54:55.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.218+0000 7f1acdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:55.199 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.218+0000 7f1acdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:55.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.583+0000 7f1ad7a3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:55.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.583+0000 7f1ad7a3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:55.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.583+0000 7f1ad7a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:54:55.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.583+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/1052193226 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ad01474b0 con 0x7f1ad00b0770 2024-03-29T10:54:55.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.584+0000 7f1acdffb640 1 -- 172.21.15.98:0/1052193226 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2920969763) 0x7f1ac4052100 con 0x7f1ad00b0770 2024-03-29T10:54:55.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.585+0000 7f1acdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:55.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.585+0000 7f1acdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:55.566 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:54:55.566 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:54:39.048768+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:39.048768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:55.570 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.588+0000 7f1ad7a3e640 10 librados: watch_flush enter 2024-03-29T10:54:55.570 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.588+0000 7f1ad7a3e640 10 librados: watch_flush exit 2024-03-29T10:54:55.570 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.588+0000 7f1ad7a3e640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-29T10:54:55.570 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.588+0000 7f1ad7a3e640 10 client.4646.objecter successfully canceled tick 2024-03-29T10:54:55.571 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.590+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/1052193226 >> v1:172.21.15.98:6816/1481799426 conn(0x7f1abc06dec0 legacy=0x7f1abc070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:55.571 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.590+0000 7f1ad7a3e640 10 monclient: shutdown 2024-03-29T10:54:55.571 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.590+0000 7f1ad7a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:55.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.590+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/1052193226 >> v1:172.21.15.98:6790/0 conn(0x7f1ad00b0770 legacy=0x7f1ad0146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:55.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.590+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/1052193226 shutdown_connections 2024-03-29T10:54:55.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.590+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/1052193226 >> 172.21.15.98:0/1052193226 conn(0x7f1ad006b660 msgr2=0x7f1ad006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:55.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.590+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/1052193226 shutdown_connections 2024-03-29T10:54:55.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.590+0000 7f1ad7a3e640 1 -- 172.21.15.98:0/1052193226 wait complete. 2024-03-29T10:54:55.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:54:55.590+0000 7f1ad7a3e640 1 librados: shutdown 2024-03-29T10:54:55.589 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:02.396 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:02.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.563+0000 7f345c767640 10 monclient: get_monmap_and_config 2024-03-29T10:55:02.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.563+0000 7f345c767640 10 monclient: build_initial_monmap 2024-03-29T10:55:02.545 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.563+0000 7f345c767640 10 monclient: monmap: 2024-03-29T10:55:02.545 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:55:02.545 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:02.545 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:55:02.564731+0000 2024-03-29T10:55:02.545 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:55:02.564731+0000 2024-03-29T10:55:02.545 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:02.545 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:02.545 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:55:02.545 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:55:02.545 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:55:02.545 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:02.546 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.564+0000 7f345c767640 1 Processor -- start 2024-03-29T10:55:02.546 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.564+0000 7f345c767640 1 -- start start 2024-03-29T10:55:02.546 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.564+0000 7f345c767640 10 monclient: init 2024-03-29T10:55:02.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f345c767640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:02.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f345c767640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:02.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f345c767640 10 monclient(hunting): picked mon.noname-c con 0x7f34540b0770 addr v1:172.21.15.189:6789/0 2024-03-29T10:55:02.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f345c767640 10 monclient(hunting): picked mon.noname-a con 0x7f345414a700 addr v1:172.21.15.98:6789/0 2024-03-29T10:55:02.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f345c767640 10 monclient(hunting): picked mon.noname-b con 0x7f34540ac5a0 addr v1:172.21.15.98:6790/0 2024-03-29T10:55:02.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f345c767640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34540aba70 con 0x7f345414a700 2024-03-29T10:55:02.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f345c767640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34540aecb0 con 0x7f34540b0770 2024-03-29T10:55:02.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f345c767640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34540af150 con 0x7f34540ac5a0 2024-03-29T10:55:02.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f345c767640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:02.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f345c767640 10 monclient(hunting): authenticate will time out at 1522.763288s 2024-03-29T10:55:02.547 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f345ad04640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f34540ac5a0 0x7f3454151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:51832/0 (socket says 172.21.15.98:51832) 2024-03-29T10:55:02.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f345ad04640 1 -- 172.21.15.98:0/3796457897 learned_addr learned my addr 172.21.15.98:0/3796457897 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:55:02.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f3459501640 1 -- 172.21.15.98:0/3796457897 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599307450 0 0) 0x7f34540aba70 con 0x7f345414a700 2024-03-29T10:55:02.548 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f3459501640 10 monclient(hunting): _negotiate 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f3459501640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f3459501640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.566+0000 7f3459501640 10 monclient(hunting): my global_id is 4584 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 1 -- 172.21.15.98:0/3796457897 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3440003520 con 0x7f345414a700 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 1 -- 172.21.15.98:0/3796457897 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139114245 0 0) 0x7f34540aecb0 con 0x7f34540b0770 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 10 monclient(hunting): _negotiate 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 10 monclient(hunting): my global_id is 4597 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 1 -- 172.21.15.98:0/3796457897 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34540aba70 con 0x7f34540b0770 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 1 -- 172.21.15.98:0/3796457897 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2319258254 0 0) 0x7f3440003520 con 0x7f345414a700 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 1 -- 172.21.15.98:0/3796457897 >> v1:172.21.15.98:6790/0 conn(0x7f34540ac5a0 legacy=0x7f3454151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 1 -- 172.21.15.98:0/3796457897 >> v1:172.21.15.189:6789/0 conn(0x7f34540b0770 legacy=0x7f3454149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.549 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 10 monclient: _finish_hunting 0 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 1 monclient: found mon.noname-a 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.98:6789/0 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 1 -- 172.21.15.98:0/3796457897 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34540ac9d0 con 0x7f345414a700 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 10 monclient: _finish_auth 0 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 10 monclient: _check_auth_tickets 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 1 -- 172.21.15.98:0/3796457897 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3444001690 con 0x7f345414a700 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f3459501640 10 monclient: dump: 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:02.550 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:02.551 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:02.551 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:02.551 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:02.551 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:02.551 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:02.551 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:02.551 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:02.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f345c767640 5 monclient: authenticate success, global_id 4584 2024-03-29T10:55:02.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.567+0000 7f345c767640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:02.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.568+0000 7f3459501640 1 -- 172.21.15.98:0/3796457897 <== mon.0 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34440017f0 con 0x7f345414a700 2024-03-29T10:55:02.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.568+0000 7f3459501640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:02.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.568+0000 7f3459501640 1 -- 172.21.15.98:0/3796457897 <== mon.0 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3444001c40 con 0x7f345414a700 2024-03-29T10:55:02.551 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.568+0000 7f3459501640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.568+0000 7f3459501640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.568+0000 7f3459501640 10 monclient: dump: 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.568+0000 7f345c767640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:02.552 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.568+0000 7f345c767640 4 set_mon_vals no callback set 2024-03-29T10:55:02.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.569+0000 7f345c767640 10 monclient: shutdown 2024-03-29T10:55:02.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.569+0000 7f345c767640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:02.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.569+0000 7f345c767640 1 -- 172.21.15.98:0/3796457897 >> v1:172.21.15.98:6789/0 conn(0x7f345414a700 legacy=0x7f34540abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.569+0000 7f345c767640 1 -- 172.21.15.98:0/3796457897 shutdown_connections 2024-03-29T10:55:02.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.569+0000 7f345c767640 1 -- 172.21.15.98:0/3796457897 >> 172.21.15.98:0/3796457897 conn(0x7f345406b660 msgr2=0x7f345406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:02.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.569+0000 7f345c767640 1 -- 172.21.15.98:0/3796457897 shutdown_connections 2024-03-29T10:55:02.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.569+0000 7f345c767640 1 -- 172.21.15.98:0/3796457897 wait complete. 2024-03-29T10:55:02.561 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.570+0000 7f345c767640 10 monclient: build_initial_monmap 2024-03-29T10:55:02.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.570+0000 7f345c767640 10 monclient: monmap: 2024-03-29T10:55:02.562 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:55:02.562 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:02.562 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:55:02.562 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:55:02.562 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:02.562 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:02.562 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:55:02.562 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:55:02.562 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:55:02.562 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:02.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.571+0000 7f345c767640 1 librados: starting msgr at 2024-03-29T10:55:02.562 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.571+0000 7f345c767640 1 librados: starting objecter 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.571+0000 7f345c767640 1 Processor -- start 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.571+0000 7f345c767640 1 -- start start 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.572+0000 7f345c767640 1 librados: setting wanted keys 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.572+0000 7f345c767640 1 librados: calling monclient init 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.572+0000 7f345c767640 10 monclient: init 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.572+0000 7f345c767640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.572+0000 7f345c767640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f345c767640 10 monclient(hunting): picked mon.noname-b con 0x7f34540ac5a0 addr v1:172.21.15.189:6789/0 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f345c767640 10 monclient(hunting): picked mon.noname-c con 0x7f34540b0770 addr v1:172.21.15.98:6790/0 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f345ad04640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f345414a700 0x7f34541e5cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41688/0 (socket says 172.21.15.98:41688) 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f345ad04640 1 -- 172.21.15.98:0/420364134 learned_addr learned my addr 172.21.15.98:0/420364134 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f345c767640 10 monclient(hunting): picked mon.noname-a con 0x7f345414a700 addr v1:172.21.15.98:6789/0 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f345c767640 1 -- 172.21.15.98:0/420364134 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3454125f90 con 0x7f345414a700 2024-03-29T10:55:02.563 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f345c767640 1 -- 172.21.15.98:0/420364134 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3454125b40 con 0x7f34540ac5a0 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f345c767640 1 -- 172.21.15.98:0/420364134 --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3454125cd0 con 0x7f34540b0770 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f345c767640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f345c767640 10 monclient(hunting): authenticate will time out at 1522.770458s 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f344affd640 10 client.?.objecter ms_handle_connect 0x7f34540b0770 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f344affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f344affd640 10 client.?.objecter ms_handle_connect 0x7f345414a700 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f344affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.573+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549199678 0 0) 0x7f3454125cd0 con 0x7f34540b0770 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.574+0000 7f344affd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.574+0000 7f344affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.574+0000 7f344affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.574+0000 7f344affd640 10 monclient(hunting): my global_id is 4652 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.574+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3438004eb0 con 0x7f34540b0770 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.574+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931117115 0 0) 0x7f3454125f90 con 0x7f345414a700 2024-03-29T10:55:02.564 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.574+0000 7f344affd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.574+0000 7f344affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.574+0000 7f344affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.574+0000 7f344affd640 10 monclient(hunting): my global_id is 4587 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.574+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3454125cd0 con 0x7f345414a700 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 client.?.objecter ms_handle_connect 0x7f34540ac5a0 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2012756484 0 0) 0x7f3438004eb0 con 0x7f34540b0770 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3454125f90 con 0x7f34540b0770 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169158261 0 0) 0x7f3454125b40 con 0x7f34540ac5a0 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 monclient(hunting): my global_id is 4600 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3438004eb0 con 0x7f34540ac5a0 2024-03-29T10:55:02.565 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3444001b00 con 0x7f34540b0770 2024-03-29T10:55:02.566 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:02.566 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:02.566 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 monclient(hunting): dump: 2024-03-29T10:55:02.566 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:02.566 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:02.566 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:02.566 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:02.566 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:02.566 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:02.566 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:02.566 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:02.566 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:02.566 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2727530754 0 0) 0x7f3454125f90 con 0x7f34540b0770 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 >> v1:172.21.15.189:6789/0 conn(0x7f34540ac5a0 legacy=0x7f34541264b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 >> v1:172.21.15.98:6789/0 conn(0x7f345414a700 legacy=0x7f34541e5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 1 monclient: found mon.c 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34541dc0d0 con 0x7f34540b0770 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 monclient: _finish_auth 0 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f344affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f345c767640 5 monclient: authenticate success, global_id 4652 2024-03-29T10:55:02.567 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f345c767640 10 monclient: _renew_subs 2024-03-29T10:55:02.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f345c767640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:02.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f345c767640 1 -- 172.21.15.98:0/420364134 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34541dc380 con 0x7f34540b0770 2024-03-29T10:55:02.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f345c767640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:02.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f345c767640 10 monclient: _renew_subs 2024-03-29T10:55:02.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f345c767640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:02.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f345c767640 1 -- 172.21.15.98:0/420364134 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34541db870 con 0x7f34540b0770 2024-03-29T10:55:02.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.575+0000 7f345c767640 1 librados: init done 2024-03-29T10:55:02.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.576+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3444001c60 con 0x7f34540b0770 2024-03-29T10:55:02.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.576+0000 7f344affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:02.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.576+0000 7f3458d00640 4 set_mon_vals no callback set 2024-03-29T10:55:02.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.577+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f3444003a40 con 0x7f34540b0770 2024-03-29T10:55:02.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.577+0000 7f344affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:02.568 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.577+0000 7f344affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:02.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.577+0000 7f344affd640 10 monclient: dump: 2024-03-29T10:55:02.569 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:02.569 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:02.569 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:02.569 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:02.569 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:02.569 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:02.569 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:02.569 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:02.569 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:02.569 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:02.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.578+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f3444019ad0 con 0x7f34540b0770 2024-03-29T10:55:02.569 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.578+0000 7f345c767640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:02.570 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.578+0000 7f345c767640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:02.570 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.578+0000 7f345c767640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:02.570 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.578+0000 7f345c767640 1 -- 172.21.15.98:0/420364134 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34540ab560 con 0x7f34540b0770 2024-03-29T10:55:02.570 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.579+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2292924465 0 0) 0x7f3444083870 con 0x7f34540b0770 2024-03-29T10:55:02.570 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.579+0000 7f344affd640 10 client.4652.objecter ms_dispatch 0x7f3454000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:55:02.570 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.579+0000 7f344affd640 3 client.4652.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:55:02.570 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.579+0000 7f345a503640 10 monclient: get_auth_request con 0x7f343806de80 auth_method 0 2024-03-29T10:55:02.571 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.580+0000 7f344affd640 3 client.4652.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:55:02.571 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.580+0000 7f344affd640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-29T10:55:02.571 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.580+0000 7f344affd640 10 client.4652.objecter ms_handle_connect 0x7f343806de80 2024-03-29T10:55:02.571 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.588+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f344404f1b0 con 0x7f34540b0770 2024-03-29T10:55:02.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.588+0000 7f344affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:02.572 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.588+0000 7f344affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:02.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.922+0000 7f345c767640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:02.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.922+0000 7f345c767640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:02.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.922+0000 7f345c767640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:02.903 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.922+0000 7f345c767640 1 -- 172.21.15.98:0/420364134 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34541282f0 con 0x7f34540b0770 2024-03-29T10:55:02.904 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.923+0000 7f344affd640 1 -- 172.21.15.98:0/420364134 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2920969763) 0x7f3444082f20 con 0x7f34540b0770 2024-03-29T10:55:02.904 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.923+0000 7f344affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:02.904 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.923+0000 7f344affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:02.905 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:55:02.905 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:54:39.048768+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:39.048768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:02.908 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.927+0000 7f345c767640 10 librados: watch_flush enter 2024-03-29T10:55:02.908 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.927+0000 7f345c767640 10 librados: watch_flush exit 2024-03-29T10:55:02.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.927+0000 7f345c767640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-29T10:55:02.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.927+0000 7f345c767640 10 client.4652.objecter successfully canceled tick 2024-03-29T10:55:02.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.927+0000 7f345c767640 1 -- 172.21.15.98:0/420364134 >> v1:172.21.15.98:6816/1481799426 conn(0x7f343806de80 legacy=0x7f3438070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.927+0000 7f345c767640 10 monclient: shutdown 2024-03-29T10:55:02.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.927+0000 7f345c767640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:02.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.927+0000 7f345c767640 1 -- 172.21.15.98:0/420364134 >> v1:172.21.15.98:6790/0 conn(0x7f34540b0770 legacy=0x7f3454126bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.927+0000 7f345c767640 1 -- 172.21.15.98:0/420364134 shutdown_connections 2024-03-29T10:55:02.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.927+0000 7f345c767640 1 -- 172.21.15.98:0/420364134 >> 172.21.15.98:0/420364134 conn(0x7f345406b660 msgr2=0x7f34540abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:02.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.927+0000 7f345c767640 1 -- 172.21.15.98:0/420364134 shutdown_connections 2024-03-29T10:55:02.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.927+0000 7f345c767640 1 -- 172.21.15.98:0/420364134 wait complete. 2024-03-29T10:55:02.909 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:02.928+0000 7f345c767640 1 librados: shutdown 2024-03-29T10:55:02.924 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:09.730 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:09.872 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.891+0000 7fa9c4fcf640 10 monclient: get_monmap_and_config 2024-03-29T10:55:09.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.891+0000 7fa9c4fcf640 10 monclient: build_initial_monmap 2024-03-29T10:55:09.873 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.891+0000 7fa9c4fcf640 10 monclient: monmap: 2024-03-29T10:55:09.873 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:55:09.873 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:09.873 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:55:09.892687+0000 2024-03-29T10:55:09.873 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:55:09.892687+0000 2024-03-29T10:55:09.873 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:09.873 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:09.873 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:55:09.873 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:55:09.873 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:55:09.873 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:09.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.892+0000 7fa9c4fcf640 1 Processor -- start 2024-03-29T10:55:09.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.892+0000 7fa9c4fcf640 1 -- start start 2024-03-29T10:55:09.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.892+0000 7fa9c4fcf640 10 monclient: init 2024-03-29T10:55:09.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.893+0000 7fa9c4fcf640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:09.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.893+0000 7fa9c4fcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:09.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.893+0000 7fa9c4fcf640 10 monclient(hunting): picked mon.noname-c con 0x7fa9c01457f0 addr v1:172.21.15.189:6789/0 2024-03-29T10:55:09.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.893+0000 7fa9c4fcf640 10 monclient(hunting): picked mon.noname-a con 0x7fa9c013f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:55:09.875 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.893+0000 7fa9c4fcf640 10 monclient(hunting): picked mon.noname-b con 0x7fa9c00a9d70 addr v1:172.21.15.98:6790/0 2024-03-29T10:55:09.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.893+0000 7fa9c4fcf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c00aaef0 con 0x7fa9c013f650 2024-03-29T10:55:09.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.893+0000 7fa9c4fcf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c0145bd0 con 0x7fa9c01457f0 2024-03-29T10:55:09.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.893+0000 7fa9c4fcf640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c0145eb0 con 0x7fa9c00a9d70 2024-03-29T10:55:09.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.893+0000 7fa9c4fcf640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:09.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.893+0000 7fa9c4fcf640 10 monclient(hunting): authenticate will time out at 1530.090961s 2024-03-29T10:55:09.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.894+0000 7fa9bed9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fa9c00a9d70 0x7fa9c014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:42402/0 (socket says 172.21.15.98:42402) 2024-03-29T10:55:09.876 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.894+0000 7fa9bed9d640 1 -- 172.21.15.98:0/3422158102 learned_addr learned my addr 172.21.15.98:0/3422158102 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:55:09.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 1 -- 172.21.15.98:0/3422158102 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027900312 0 0) 0x7fa9c0145eb0 con 0x7fa9c00a9d70 2024-03-29T10:55:09.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 10 monclient(hunting): my global_id is 4655 2024-03-29T10:55:09.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 1 -- 172.21.15.98:0/3422158102 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a8003520 con 0x7fa9c00a9d70 2024-03-29T10:55:09.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 1 -- 172.21.15.98:0/3422158102 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257084533 0 0) 0x7fa9c00aaef0 con 0x7fa9c013f650 2024-03-29T10:55:09.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 10 monclient(hunting): my global_id is 4590 2024-03-29T10:55:09.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 1 -- 172.21.15.98:0/3422158102 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c0145eb0 con 0x7fa9c013f650 2024-03-29T10:55:09.878 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 1 -- 172.21.15.98:0/3422158102 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206908609 0 0) 0x7fa9c0145bd0 con 0x7fa9c01457f0 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 10 monclient(hunting): my global_id is 4603 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.895+0000 7fa9bd59a640 1 -- 172.21.15.98:0/3422158102 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c00aaef0 con 0x7fa9c01457f0 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 1 -- 172.21.15.98:0/3422158102 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 320221749 0 0) 0x7fa9a8003520 con 0x7fa9c00a9d70 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 1 -- 172.21.15.98:0/3422158102 >> v1:172.21.15.189:6789/0 conn(0x7fa9c01457f0 legacy=0x7fa9c013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 1 -- 172.21.15.98:0/3422158102 >> v1:172.21.15.98:6789/0 conn(0x7fa9c013f650 legacy=0x7fa9c00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 10 monclient: _finish_hunting 0 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 1 monclient: found mon.noname-b 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 1 -- 172.21.15.98:0/3422158102 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c0146340 con 0x7fa9c00a9d70 2024-03-29T10:55:09.879 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 10 monclient: _finish_auth 0 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 10 monclient: _check_auth_tickets 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9c4fcf640 5 monclient: authenticate success, global_id 4655 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9c4fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 1 -- 172.21.15.98:0/3422158102 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa9b80019d0 con 0x7fa9c00a9d70 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 10 monclient: dump: 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:09.880 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 1 -- 172.21.15.98:0/3422158102 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9b8001c20 con 0x7fa9c00a9d70 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 1 -- 172.21.15.98:0/3422158102 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa9b8001ff0 con 0x7fa9c00a9d70 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9bd59a640 10 monclient: dump: 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:09.881 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:09.882 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:09.882 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:09.882 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:09.882 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:09.882 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:09.882 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9c4fcf640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:09.882 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.896+0000 7fa9c4fcf640 4 set_mon_vals no callback set 2024-03-29T10:55:09.882 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.897+0000 7fa9c4fcf640 10 monclient: shutdown 2024-03-29T10:55:09.882 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.897+0000 7fa9c4fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:09.882 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.897+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/3422158102 >> v1:172.21.15.98:6790/0 conn(0x7fa9c00a9d70 legacy=0x7fa9c014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.886 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.897+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/3422158102 shutdown_connections 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.897+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/3422158102 >> 172.21.15.98:0/3422158102 conn(0x7fa9c0066a00 msgr2=0x7fa9c0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.897+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/3422158102 shutdown_connections 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.897+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/3422158102 wait complete. 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.898+0000 7fa9c4fcf640 10 monclient: build_initial_monmap 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.898+0000 7fa9c4fcf640 10 monclient: monmap: 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:55:09.887 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.899+0000 7fa9c4fcf640 1 librados: starting msgr at 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.899+0000 7fa9c4fcf640 1 librados: starting objecter 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.899+0000 7fa9c4fcf640 1 Processor -- start 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.899+0000 7fa9c4fcf640 1 -- start start 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.899+0000 7fa9c4fcf640 1 librados: setting wanted keys 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.899+0000 7fa9c4fcf640 1 librados: calling monclient init 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.899+0000 7fa9c4fcf640 10 monclient: init 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.900+0000 7fa9c4fcf640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.900+0000 7fa9c4fcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.900+0000 7fa9c4fcf640 10 monclient(hunting): picked mon.noname-b con 0x7fa9c00a9d70 addr v1:172.21.15.189:6789/0 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.900+0000 7fa9c4fcf640 10 monclient(hunting): picked mon.noname-a con 0x7fa9c013f650 addr v1:172.21.15.98:6789/0 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.900+0000 7fa9c4fcf640 10 monclient(hunting): picked mon.noname-c con 0x7fa9c01457f0 addr v1:172.21.15.98:6790/0 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.900+0000 7fa9c4fcf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c0070c10 con 0x7fa9c013f650 2024-03-29T10:55:09.888 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.900+0000 7fa9c4fcf640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c006f8f0 con 0x7fa9c00a9d70 2024-03-29T10:55:09.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.900+0000 7fa9c4fcf640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c006fa80 con 0x7fa9c01457f0 2024-03-29T10:55:09.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.900+0000 7fa9c4fcf640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:09.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.900+0000 7fa9c4fcf640 10 monclient(hunting): authenticate will time out at 1530.097614s 2024-03-29T10:55:09.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.900+0000 7fa9bed9d640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7fa9c01457f0 0x7fa9c01ded20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:42684/0 (socket says 172.21.15.98:42684) 2024-03-29T10:55:09.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.900+0000 7fa9bed9d640 1 -- 172.21.15.98:0/2045252352 learned_addr learned my addr 172.21.15.98:0/2045252352 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:55:09.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.901+0000 7fa9aeffd640 10 client.?.objecter ms_handle_connect 0x7fa9c01457f0 2024-03-29T10:55:09.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.901+0000 7fa9aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:09.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.901+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809540835 0 0) 0x7fa9c006fa80 con 0x7fa9c01457f0 2024-03-29T10:55:09.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.901+0000 7fa9aeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.889 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.901+0000 7fa9aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.901+0000 7fa9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.901+0000 7fa9aeffd640 10 monclient(hunting): my global_id is 4658 2024-03-29T10:55:09.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.901+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a0004eb0 con 0x7fa9c01457f0 2024-03-29T10:55:09.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.901+0000 7fa9aeffd640 10 client.?.objecter ms_handle_connect 0x7fa9c013f650 2024-03-29T10:55:09.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.901+0000 7fa9aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:09.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.901+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2601478402 0 0) 0x7fa9a0004eb0 con 0x7fa9c01457f0 2024-03-29T10:55:09.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9c006fa80 con 0x7fa9c01457f0 2024-03-29T10:55:09.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 client.?.objecter ms_handle_connect 0x7fa9c00a9d70 2024-03-29T10:55:09.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:09.890 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956734120 0 0) 0x7fa9c0070c10 con 0x7fa9c013f650 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient(hunting): my global_id is 4593 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a0004eb0 con 0x7fa9c013f650 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa9b8001f30 con 0x7fa9c01457f0 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient(hunting): dump: 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:09.891 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687999494 0 0) 0x7fa9c006f8f0 con 0x7fa9c00a9d70 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient(hunting): my global_id is 4606 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c0070c10 con 0x7fa9c00a9d70 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3745182554 0 0) 0x7fa9c006fa80 con 0x7fa9c01457f0 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 >> v1:172.21.15.189:6789/0 conn(0x7fa9c00a9d70 legacy=0x7fa9c00703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 >> v1:172.21.15.98:6789/0 conn(0x7fa9c013f650 legacy=0x7fa9c0139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.892 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 1 monclient: found mon.c 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c013a470 con 0x7fa9c01457f0 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9c4fcf640 5 monclient: authenticate success, global_id 4658 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9c4fcf640 10 monclient: _renew_subs 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9c4fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/2045252352 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c013a720 con 0x7fa9c01457f0 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9c4fcf640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9c4fcf640 10 monclient: _renew_subs 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9c4fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:09.893 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/2045252352 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c0142450 con 0x7fa9c01457f0 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.902+0000 7fa9c4fcf640 1 librados: init done 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.903+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9b8004fe0 con 0x7fa9c01457f0 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.903+0000 7fa9aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.903+0000 7fa9bcd99640 4 set_mon_vals no callback set 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.904+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7fa9b8005210 con 0x7fa9c01457f0 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.904+0000 7fa9aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.904+0000 7fa9aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.904+0000 7fa9aeffd640 10 monclient: dump: 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:09.894 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:09.895 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:09.895 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:09.895 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:09.895 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:09.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.904+0000 7fa9c4fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:09.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.904+0000 7fa9c4fcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:09.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.904+0000 7fa9c4fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:09.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.904+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/2045252352 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a4005080 con 0x7fa9c01457f0 2024-03-29T10:55:09.895 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.906+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7fa9b8005370 con 0x7fa9c01457f0 2024-03-29T10:55:09.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.906+0000 7fa9be59c640 10 monclient: get_auth_request con 0x7fa9a006de20 auth_method 0 2024-03-29T10:55:09.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.907+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2292924465 0 0) 0x7fa9b8084e10 con 0x7fa9c01457f0 2024-03-29T10:55:09.896 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.907+0000 7fa9aeffd640 10 client.4658.objecter ms_dispatch 0x7fa9c0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:55:09.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.907+0000 7fa9aeffd640 3 client.4658.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:55:09.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.907+0000 7fa9aeffd640 3 client.4658.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:55:09.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.907+0000 7fa9aeffd640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-29T10:55:09.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.907+0000 7fa9aeffd640 10 client.4658.objecter ms_handle_connect 0x7fa9a006de20 2024-03-29T10:55:09.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.914+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9b8050750 con 0x7fa9c01457f0 2024-03-29T10:55:09.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.914+0000 7fa9aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:09.897 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:09.914+0000 7fa9aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:10.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.287+0000 7fa9c4fcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:10.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.287+0000 7fa9c4fcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:10.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.287+0000 7fa9c4fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:10.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.287+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/2045252352 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9a4005330 con 0x7fa9c01457f0 2024-03-29T10:55:10.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.288+0000 7fa9aeffd640 1 -- 172.21.15.98:0/2045252352 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2920969763) 0x7fa9b8052680 con 0x7fa9c01457f0 2024-03-29T10:55:10.268 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.288+0000 7fa9aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:10.269 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.288+0000 7fa9aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:10.269 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:55:10.269 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:54:39.048768+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:39.048768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:10.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.291+0000 7fa9c4fcf640 10 librados: watch_flush enter 2024-03-29T10:55:10.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.291+0000 7fa9c4fcf640 10 librados: watch_flush exit 2024-03-29T10:55:10.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.291+0000 7fa9c4fcf640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-29T10:55:10.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.291+0000 7fa9c4fcf640 10 client.4658.objecter successfully canceled tick 2024-03-29T10:55:10.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.291+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/2045252352 >> v1:172.21.15.98:6816/1481799426 conn(0x7fa9a006de20 legacy=0x7fa9a00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:10.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.291+0000 7fa9c4fcf640 10 monclient: shutdown 2024-03-29T10:55:10.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.291+0000 7fa9c4fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:10.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.291+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/2045252352 >> v1:172.21.15.98:6790/0 conn(0x7fa9c01457f0 legacy=0x7fa9c01ded20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:10.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.292+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/2045252352 shutdown_connections 2024-03-29T10:55:10.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.292+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/2045252352 >> 172.21.15.98:0/2045252352 conn(0x7fa9c0066a00 msgr2=0x7fa9c013ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:10.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.292+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/2045252352 shutdown_connections 2024-03-29T10:55:10.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.292+0000 7fa9c4fcf640 1 -- 172.21.15.98:0/2045252352 wait complete. 2024-03-29T10:55:10.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:10.292+0000 7fa9c4fcf640 1 librados: shutdown 2024-03-29T10:55:10.288 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:17.095 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:17.248 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.265+0000 7f53561a5640 10 monclient: get_monmap_and_config 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.266+0000 7f53561a5640 10 monclient: build_initial_monmap 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.266+0000 7f53561a5640 10 monclient: monmap: 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:55:17.266870+0000 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:55:17.266870+0000 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.267+0000 7f53561a5640 1 Processor -- start 2024-03-29T10:55:17.249 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.267+0000 7f53561a5640 1 -- start start 2024-03-29T10:55:17.250 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.267+0000 7f53561a5640 10 monclient: init 2024-03-29T10:55:17.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.268+0000 7f53561a5640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:17.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.268+0000 7f53561a5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:17.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.268+0000 7f53561a5640 10 monclient(hunting): picked mon.noname-a con 0x7f53500ad6c0 addr v1:172.21.15.98:6789/0 2024-03-29T10:55:17.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.268+0000 7f53561a5640 10 monclient(hunting): picked mon.noname-b con 0x7f5350152980 addr v1:172.21.15.98:6790/0 2024-03-29T10:55:17.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.268+0000 7f53561a5640 10 monclient(hunting): picked mon.noname-c con 0x7f53501588d0 addr v1:172.21.15.189:6789/0 2024-03-29T10:55:17.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.268+0000 7f53561a5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53500a94e0 con 0x7f53500ad6c0 2024-03-29T10:55:17.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.268+0000 7f53561a5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53500af3a0 con 0x7f53501588d0 2024-03-29T10:55:17.251 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.268+0000 7f53561a5640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53500afb60 con 0x7f5350152980 2024-03-29T10:55:17.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.268+0000 7f53561a5640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:17.252 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.268+0000 7f53561a5640 10 monclient(hunting): authenticate will time out at 1537.465572s 2024-03-29T10:55:17.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.273+0000 7f534effd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f5350152980 0x7f5350152df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:51481/0 (socket says 172.21.15.98:51481) 2024-03-29T10:55:17.254 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.273+0000 7f534effd640 1 -- 172.21.15.98:0/664174037 learned_addr learned my addr 172.21.15.98:0/664174037 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:55:17.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.273+0000 7f534e7fc640 1 -- 172.21.15.98:0/664174037 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638348455 0 0) 0x7f53500afb60 con 0x7f5350152980 2024-03-29T10:55:17.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.273+0000 7f534e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:17.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.273+0000 7f534e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:17.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.273+0000 7f534e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:17.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.273+0000 7f534e7fc640 10 monclient(hunting): my global_id is 4661 2024-03-29T10:55:17.255 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.273+0000 7f534e7fc640 1 -- 172.21.15.98:0/664174037 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5338003520 con 0x7f5350152980 2024-03-29T10:55:17.257 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 1 -- 172.21.15.98:0/664174037 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 380675106 0 0) 0x7f5338003520 con 0x7f5350152980 2024-03-29T10:55:17.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 1 -- 172.21.15.98:0/664174037 >> v1:172.21.15.189:6789/0 conn(0x7f53501588d0 legacy=0x7f535015acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:17.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 1 -- 172.21.15.98:0/664174037 >> v1:172.21.15.98:6789/0 conn(0x7f53500ad6c0 legacy=0x7f535014fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:17.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:17.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 1 monclient: found mon.noname-b 2024-03-29T10:55:17.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:17.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:55:17.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 1 -- 172.21.15.98:0/664174037 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53500aab80 con 0x7f5350152980 2024-03-29T10:55:17.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:17.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:17.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:17.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f53561a5640 5 monclient: authenticate success, global_id 4661 2024-03-29T10:55:17.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f53561a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:17.258 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 1 -- 172.21.15.98:0/664174037 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5348001a10 con 0x7f5350152980 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f534e7fc640 10 monclient: dump: 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.275+0000 7f53561a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:17.259 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.277+0000 7f534e7fc640 1 -- 172.21.15.98:0/664174037 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5348001ca0 con 0x7f5350152980 2024-03-29T10:55:17.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.277+0000 7f534e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:17.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.277+0000 7f53561a5640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:17.260 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.277+0000 7f53561a5640 4 set_mon_vals no callback set 2024-03-29T10:55:17.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.277+0000 7f534e7fc640 1 -- 172.21.15.98:0/664174037 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5348002030 con 0x7f5350152980 2024-03-29T10:55:17.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.277+0000 7f534e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:17.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.277+0000 7f534e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:17.262 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.277+0000 7f534e7fc640 10 monclient: dump: 2024-03-29T10:55:17.262 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:17.262 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:17.262 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:17.262 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:17.262 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:17.262 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:17.262 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:17.262 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.277+0000 7f53561a5640 10 monclient: shutdown 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.277+0000 7f53561a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.277+0000 7f53561a5640 1 -- 172.21.15.98:0/664174037 >> v1:172.21.15.98:6790/0 conn(0x7f5350152980 legacy=0x7f5350152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.278+0000 7f53561a5640 1 -- 172.21.15.98:0/664174037 shutdown_connections 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.278+0000 7f53561a5640 1 -- 172.21.15.98:0/664174037 >> 172.21.15.98:0/664174037 conn(0x7f53500b35e0 msgr2=0x7f53500b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.278+0000 7f53561a5640 1 -- 172.21.15.98:0/664174037 shutdown_connections 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.278+0000 7f53561a5640 1 -- 172.21.15.98:0/664174037 wait complete. 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.279+0000 7f53561a5640 10 monclient: build_initial_monmap 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.279+0000 7f53561a5640 10 monclient: monmap: 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:55:17.263 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:17.264 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:17.264 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:55:17.264 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:55:17.264 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:55:17.264 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:17.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.279+0000 7f53561a5640 1 librados: starting msgr at 2024-03-29T10:55:17.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.279+0000 7f53561a5640 1 librados: starting objecter 2024-03-29T10:55:17.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.279+0000 7f53561a5640 1 Processor -- start 2024-03-29T10:55:17.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.280+0000 7f53561a5640 1 -- start start 2024-03-29T10:55:17.264 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.280+0000 7f53561a5640 1 librados: setting wanted keys 2024-03-29T10:55:17.266 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.280+0000 7f53561a5640 1 librados: calling monclient init 2024-03-29T10:55:17.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.280+0000 7f53561a5640 10 monclient: init 2024-03-29T10:55:17.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.280+0000 7f53561a5640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:17.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.280+0000 7f53561a5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:55:17.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.282+0000 7f53561a5640 10 monclient(hunting): picked mon.noname-b con 0x7f53500ad6c0 addr v1:172.21.15.189:6789/0 2024-03-29T10:55:17.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.284+0000 7f53561a5640 10 monclient(hunting): picked mon.noname-c con 0x7f5350152980 addr v1:172.21.15.98:6790/0 2024-03-29T10:55:17.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.284+0000 7f53561a5640 10 monclient(hunting): picked mon.noname-a con 0x7f53501588d0 addr v1:172.21.15.98:6789/0 2024-03-29T10:55:17.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.284+0000 7f53561a5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535014d900 con 0x7f53501588d0 2024-03-29T10:55:17.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f53561a5640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5350146910 con 0x7f53500ad6c0 2024-03-29T10:55:17.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f53561a5640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535014d590 con 0x7f5350152980 2024-03-29T10:55:17.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f53561a5640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:17.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f53561a5640 10 monclient(hunting): authenticate will time out at 1537.482001s 2024-03-29T10:55:17.267 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f534effd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f5350152980 0x7f53501e4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:51483/0 (socket says 172.21.15.98:51483) 2024-03-29T10:55:17.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f534effd640 1 -- 172.21.15.98:0/3146827189 learned_addr learned my addr 172.21.15.98:0/3146827189 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:55:17.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f5337fff640 10 client.?.objecter ms_handle_connect 0x7f5350152980 2024-03-29T10:55:17.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f5337fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:17.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917344430 0 0) 0x7f535014d590 con 0x7f5350152980 2024-03-29T10:55:17.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f5337fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:17.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f5337fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:17.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f5337fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:17.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f5337fff640 10 monclient(hunting): my global_id is 4664 2024-03-29T10:55:17.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.285+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f533c004eb0 con 0x7f5350152980 2024-03-29T10:55:17.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.286+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3257309880 0 0) 0x7f533c004eb0 con 0x7f5350152980 2024-03-29T10:55:17.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.286+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f535014d590 con 0x7f5350152980 2024-03-29T10:55:17.271 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.286+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f5348001990 con 0x7f5350152980 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.286+0000 7f5337fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.286+0000 7f5337fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.286+0000 7f5337fff640 10 monclient(hunting): dump: 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.286+0000 7f5337fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:17.272 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.288+0000 7f5337fff640 10 client.?.objecter ms_handle_connect 0x7f53501588d0 2024-03-29T10:55:17.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.288+0000 7f5337fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:17.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.288+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100132380 0 0) 0x7f535014d900 con 0x7f53501588d0 2024-03-29T10:55:17.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.288+0000 7f5337fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:17.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.289+0000 7f5337fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:17.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.289+0000 7f5337fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:17.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.289+0000 7f5337fff640 10 monclient(hunting): my global_id is 4596 2024-03-29T10:55:17.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.289+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f533c004eb0 con 0x7f53501588d0 2024-03-29T10:55:17.273 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.289+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1425555635 0 0) 0x7f535014d590 con 0x7f5350152980 2024-03-29T10:55:17.274 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 >> v1:172.21.15.189:6789/0 conn(0x7f53500ad6c0 legacy=0x7f53500b2630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:17.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 >> v1:172.21.15.98:6789/0 conn(0x7f53501588d0 legacy=0x7f53501e4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:17.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f5337fff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:17.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f5337fff640 1 monclient: found mon.c 2024-03-29T10:55:17.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f5337fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:17.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f5337fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:17.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535014c760 con 0x7f5350152980 2024-03-29T10:55:17.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f5337fff640 10 monclient: _finish_auth 0 2024-03-29T10:55:17.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f5337fff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:17.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f5337fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:17.278 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f53561a5640 5 monclient: authenticate success, global_id 4664 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f53561a5640 10 monclient: _renew_subs 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f53561a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f53561a5640 1 -- 172.21.15.98:0/3146827189 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535014c970 con 0x7f5350152980 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f53561a5640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f53561a5640 10 monclient: _renew_subs 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f53561a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f53561a5640 1 -- 172.21.15.98:0/3146827189 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53500b2f20 con 0x7f5350152980 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.290+0000 7f53561a5640 1 librados: init done 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.291+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5348001af0 con 0x7f5350152980 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.291+0000 7f5337fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.291+0000 7f534d7fa640 4 set_mon_vals no callback set 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.292+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f53480051a0 con 0x7f5350152980 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.292+0000 7f5337fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.292+0000 7f5337fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:17.279 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.292+0000 7f5337fff640 10 monclient: dump: 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.292+0000 7f53561a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.292+0000 7f53561a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.292+0000 7f53561a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.292+0000 7f53561a5640 1 -- 172.21.15.98:0/3146827189 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5330005080 con 0x7f5350152980 2024-03-29T10:55:17.280 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.296+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f534801b170 con 0x7f5350152980 2024-03-29T10:55:17.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.297+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2292924465 0 0) 0x7f5348084fa0 con 0x7f5350152980 2024-03-29T10:55:17.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.297+0000 7f5337fff640 10 client.4664.objecter ms_dispatch 0x7f5350000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:55:17.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.297+0000 7f5337fff640 3 client.4664.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:55:17.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.297+0000 7f5337fff640 3 client.4664.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:55:17.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.297+0000 7f5337fff640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-29T10:55:17.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.299+0000 7f534f7fe640 10 monclient: get_auth_request con 0x7f533c06db30 auth_method 0 2024-03-29T10:55:17.281 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.299+0000 7f5337fff640 10 client.4664.objecter ms_handle_connect 0x7f533c06db30 2024-03-29T10:55:17.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.303+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53480506b0 con 0x7f5350152980 2024-03-29T10:55:17.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.303+0000 7f5337fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:17.284 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.303+0000 7f5337fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.623+0000 7f53561a5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.623+0000 7f53561a5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.623+0000 7f53561a5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:17.604 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.623+0000 7f53561a5640 1 -- 172.21.15.98:0/3146827189 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5330005330 con 0x7f5350152980 2024-03-29T10:55:17.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.624+0000 7f5337fff640 1 -- 172.21.15.98:0/3146827189 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2920969763) 0x7f53480527b0 con 0x7f5350152980 2024-03-29T10:55:17.605 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.624+0000 7f5337fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:17.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.625+0000 7f5337fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:17.606 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:55:17.607 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:54:39.048768+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:39.048768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:17.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.629+0000 7f53561a5640 10 librados: watch_flush enter 2024-03-29T10:55:17.612 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.629+0000 7f53561a5640 10 librados: watch_flush exit 2024-03-29T10:55:17.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.629+0000 7f53561a5640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-29T10:55:17.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.629+0000 7f53561a5640 10 client.4664.objecter successfully canceled tick 2024-03-29T10:55:17.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.629+0000 7f53561a5640 1 -- 172.21.15.98:0/3146827189 >> v1:172.21.15.98:6816/1481799426 conn(0x7f533c06db30 legacy=0x7f533c06fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:17.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.629+0000 7f53561a5640 10 monclient: shutdown 2024-03-29T10:55:17.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.629+0000 7f53561a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:17.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.629+0000 7f53561a5640 1 -- 172.21.15.98:0/3146827189 >> v1:172.21.15.98:6790/0 conn(0x7f5350152980 legacy=0x7f53501e4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:17.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.629+0000 7f53561a5640 1 -- 172.21.15.98:0/3146827189 shutdown_connections 2024-03-29T10:55:17.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.629+0000 7f53561a5640 1 -- 172.21.15.98:0/3146827189 >> 172.21.15.98:0/3146827189 conn(0x7f53500b35e0 msgr2=0x7f53500ab380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:17.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.630+0000 7f53561a5640 1 -- 172.21.15.98:0/3146827189 shutdown_connections 2024-03-29T10:55:17.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.630+0000 7f53561a5640 1 -- 172.21.15.98:0/3146827189 wait complete. 2024-03-29T10:55:17.613 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:17.632+0000 7f53561a5640 1 librados: shutdown 2024-03-29T10:55:17.630 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:24.437 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:24.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.625+0000 7f13c8832640 10 monclient: get_monmap_and_config 2024-03-29T10:55:24.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.625+0000 7f13c8832640 10 monclient: build_initial_monmap 2024-03-29T10:55:24.606 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.625+0000 7f13c8832640 10 monclient: monmap: 2024-03-29T10:55:24.606 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:55:24.606 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:24.606 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:55:24.625934+0000 2024-03-29T10:55:24.606 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:55:24.625934+0000 2024-03-29T10:55:24.606 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:24.606 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:24.607 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:55:24.607 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:55:24.607 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:55:24.607 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:24.607 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.626+0000 7f13c8832640 1 Processor -- start 2024-03-29T10:55:24.608 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.626+0000 7f13c8832640 1 -- start start 2024-03-29T10:55:24.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.626+0000 7f13c8832640 10 monclient: init 2024-03-29T10:55:24.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.627+0000 7f13c8832640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:24.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.627+0000 7f13c8832640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:24.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.627+0000 7f13c8832640 10 monclient(hunting): picked mon.noname-c con 0x7f13bc006a70 addr v1:172.21.15.189:6789/0 2024-03-29T10:55:24.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.627+0000 7f13c8832640 10 monclient(hunting): picked mon.noname-a con 0x7f13bc02b100 addr v1:172.21.15.98:6789/0 2024-03-29T10:55:24.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.628+0000 7f13c8832640 10 monclient(hunting): picked mon.noname-b con 0x7f13bc029c30 addr v1:172.21.15.98:6790/0 2024-03-29T10:55:24.609 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.628+0000 7f13c8832640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc02a270 con 0x7f13bc02b100 2024-03-29T10:55:24.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.628+0000 7f13c8832640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc02c440 con 0x7f13bc006a70 2024-03-29T10:55:24.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.628+0000 7f13c8832640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0c9440 con 0x7f13bc029c30 2024-03-29T10:55:24.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.628+0000 7f13c8832640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:24.610 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.628+0000 7f13c8832640 10 monclient(hunting): authenticate will time out at 1544.825400s 2024-03-29T10:55:24.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.629+0000 7f13c359e640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f13bc029c30 0x7f13bc0d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:43570/0 (socket says 172.21.15.98:43570) 2024-03-29T10:55:24.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.629+0000 7f13c359e640 1 -- 172.21.15.98:0/3738085281 learned_addr learned my addr 172.21.15.98:0/3738085281 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:55:24.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.631+0000 7f13c1d9b640 1 -- 172.21.15.98:0/3738085281 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065721875 0 0) 0x7f13bc0c9440 con 0x7f13bc029c30 2024-03-29T10:55:24.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.631+0000 7f13c1d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:55:24.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.631+0000 7f13c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:24.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.631+0000 7f13c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:24.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.631+0000 7f13c1d9b640 10 monclient(hunting): my global_id is 4667 2024-03-29T10:55:24.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.631+0000 7f13c1d9b640 1 -- 172.21.15.98:0/3738085281 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ac003520 con 0x7f13bc029c30 2024-03-29T10:55:24.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 1 -- 172.21.15.98:0/3738085281 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2420741592 0 0) 0x7f13ac003520 con 0x7f13bc029c30 2024-03-29T10:55:24.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 1 -- 172.21.15.98:0/3738085281 >> v1:172.21.15.189:6789/0 conn(0x7f13bc006a70 legacy=0x7f13bc0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:24.616 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 1 -- 172.21.15.98:0/3738085281 >> v1:172.21.15.98:6789/0 conn(0x7f13bc02b100 legacy=0x7f13bc0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 1 monclient: found mon.noname-b 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 1 -- 172.21.15.98:0/3738085281 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0c95d0 con 0x7f13bc029c30 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 10 monclient: _finish_auth 0 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 1 -- 172.21.15.98:0/3738085281 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f13c404f730 con 0x7f13bc029c30 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 10 monclient: dump: 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:24.617 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 1 -- 172.21.15.98:0/3738085281 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13c4075f10 con 0x7f13bc029c30 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 1 -- 172.21.15.98:0/3738085281 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f13c40758b0 con 0x7f13bc029c30 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.632+0000 7f13c1d9b640 10 monclient: dump: 2024-03-29T10:55:24.618 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.633+0000 7f13c8832640 5 monclient: authenticate success, global_id 4667 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.633+0000 7f13c8832640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.633+0000 7f13c8832640 4 set_mon_vals no callback set 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.634+0000 7f13c8832640 10 monclient: shutdown 2024-03-29T10:55:24.619 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.634+0000 7f13c8832640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:24.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.634+0000 7f13c8832640 1 -- 172.21.15.98:0/3738085281 >> v1:172.21.15.98:6790/0 conn(0x7f13bc029c30 legacy=0x7f13bc0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:24.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.634+0000 7f13c8832640 1 -- 172.21.15.98:0/3738085281 shutdown_connections 2024-03-29T10:55:24.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.634+0000 7f13c8832640 1 -- 172.21.15.98:0/3738085281 >> 172.21.15.98:0/3738085281 conn(0x7f13bc0058f0 msgr2=0x7f13bc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:24.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.634+0000 7f13c8832640 1 -- 172.21.15.98:0/3738085281 shutdown_connections 2024-03-29T10:55:24.620 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.634+0000 7f13c8832640 1 -- 172.21.15.98:0/3738085281 wait complete. 2024-03-29T10:55:24.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.635+0000 7f13c8832640 10 monclient: build_initial_monmap 2024-03-29T10:55:24.623 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.635+0000 7f13c8832640 10 monclient: monmap: 2024-03-29T10:55:24.623 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:55:24.623 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:24.623 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:55:24.623 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:55:24.623 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.636+0000 7f13c8832640 1 librados: starting msgr at 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.636+0000 7f13c8832640 1 librados: starting objecter 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.636+0000 7f13c8832640 1 Processor -- start 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.636+0000 7f13c8832640 1 -- start start 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.636+0000 7f13c8832640 1 librados: setting wanted keys 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.636+0000 7f13c8832640 1 librados: calling monclient init 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.636+0000 7f13c8832640 10 monclient: init 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.637+0000 7f13c8832640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:24.624 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.637+0000 7f13c8832640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.640+0000 7f13c8832640 10 monclient(hunting): picked mon.noname-a con 0x7f13bc006a70 addr v1:172.21.15.98:6789/0 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.640+0000 7f13c8832640 10 monclient(hunting): picked mon.noname-b con 0x7f13bc029c30 addr v1:172.21.15.189:6789/0 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.640+0000 7f13c8832640 10 monclient(hunting): picked mon.noname-c con 0x7f13bc0c9980 addr v1:172.21.15.98:6790/0 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.640+0000 7f13c8832640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0cb650 con 0x7f13bc006a70 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.640+0000 7f13c8832640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0cbae0 con 0x7f13bc029c30 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.640+0000 7f13c8832640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0cb0a0 con 0x7f13bc0c9980 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.640+0000 7f13c8832640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.640+0000 7f13c8832640 10 monclient(hunting): authenticate will time out at 1544.837470s 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.640+0000 7f13c2d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f13bc006a70 0x7f13bc0ca740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57865/0 (socket says 172.21.15.98:57865) 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.640+0000 7f13c2d9d640 1 -- 172.21.15.98:0/2405385773 learned_addr learned my addr 172.21.15.98:0/2405385773 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 10 client.?.objecter ms_handle_connect 0x7f13bc006a70 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 10 client.?.objecter ms_handle_connect 0x7f13bc0c9980 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:24.625 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171665569 0 0) 0x7f13bc0cb0a0 con 0x7f13bc0c9980 2024-03-29T10:55:24.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:24.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:24.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:24.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 10 monclient(hunting): my global_id is 4670 2024-03-29T10:55:24.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b0004eb0 con 0x7f13bc0c9980 2024-03-29T10:55:24.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962378317 0 0) 0x7f13bc0cb650 con 0x7f13bc006a70 2024-03-29T10:55:24.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:24.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:24.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:24.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.641+0000 7f13ab7fe640 10 monclient(hunting): my global_id is 4599 2024-03-29T10:55:24.626 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.642+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc0cb0a0 con 0x7f13bc006a70 2024-03-29T10:55:24.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.643+0000 7f13ab7fe640 10 client.?.objecter ms_handle_connect 0x7f13bc029c30 2024-03-29T10:55:24.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.643+0000 7f13ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:24.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.643+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3508662273 0 0) 0x7f13b0004eb0 con 0x7f13bc0c9980 2024-03-29T10:55:24.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.643+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13bc0cb650 con 0x7f13bc0c9980 2024-03-29T10:55:24.627 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 716382939 0 0) 0x7f13bc0cb0a0 con 0x7f13bc006a70 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13b0004eb0 con 0x7f13bc006a70 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f13c404f400 con 0x7f13bc0c9980 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:24.628 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f13b4001870 con 0x7f13bc006a70 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient(hunting): dump: 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:24.629 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850253737 0 0) 0x7f13bc0cbae0 con 0x7f13bc029c30 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient(hunting): my global_id is 4609 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc0cb0a0 con 0x7f13bc029c30 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1532188312 0 0) 0x7f13bc0cb650 con 0x7f13bc0c9980 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 >> v1:172.21.15.189:6789/0 conn(0x7f13bc029c30 legacy=0x7f13bc0bf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 >> v1:172.21.15.98:6789/0 conn(0x7f13bc006a70 legacy=0x7f13bc0ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 1 monclient: found mon.c 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:24.630 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0cb230 con 0x7f13bc0c9980 2024-03-29T10:55:24.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:24.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:24.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:24.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13c8832640 5 monclient: authenticate success, global_id 4670 2024-03-29T10:55:24.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13c8832640 10 monclient: _renew_subs 2024-03-29T10:55:24.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13c8832640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:24.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13c8832640 1 -- 172.21.15.98:0/2405385773 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc0bfe90 con 0x7f13bc0c9980 2024-03-29T10:55:24.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13c8832640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:24.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13c8832640 10 monclient: _renew_subs 2024-03-29T10:55:24.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13c8832640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:24.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13c8832640 1 -- 172.21.15.98:0/2405385773 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc0cda60 con 0x7f13bc0c9980 2024-03-29T10:55:24.631 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13c8832640 1 librados: init done 2024-03-29T10:55:24.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13c404f730 con 0x7f13bc0c9980 2024-03-29T10:55:24.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:24.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.644+0000 7f13c159a640 4 set_mon_vals no callback set 2024-03-29T10:55:24.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.646+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f13c4067d90 con 0x7f13bc0c9980 2024-03-29T10:55:24.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.646+0000 7f13ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:24.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.646+0000 7f13ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:24.632 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.646+0000 7f13ab7fe640 10 monclient: dump: 2024-03-29T10:55:24.632 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:24.632 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:24.632 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:24.632 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:24.632 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:24.632 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:24.633 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:24.633 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:24.633 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:24.633 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:24.633 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.646+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f13c4067fb0 con 0x7f13bc0c9980 2024-03-29T10:55:24.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.647+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2292924465 0 0) 0x7f13c40ff310 con 0x7f13bc0c9980 2024-03-29T10:55:24.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.647+0000 7f13ab7fe640 10 client.4670.objecter ms_dispatch 0x7f13c4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:55:24.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.647+0000 7f13ab7fe640 3 client.4670.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:55:24.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.647+0000 7f13ab7fe640 3 client.4670.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:55:24.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.647+0000 7f13c2d9d640 10 monclient: get_auth_request con 0x7f13b006df60 auth_method 0 2024-03-29T10:55:24.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.647+0000 7f13ab7fe640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-29T10:55:24.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.647+0000 7f13ab7fe640 10 client.4670.objecter ms_handle_connect 0x7f13b006df60 2024-03-29T10:55:24.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.650+0000 7f13c8832640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:24.634 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.650+0000 7f13c8832640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:24.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.650+0000 7f13c8832640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:24.635 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.650+0000 7f13c8832640 1 -- 172.21.15.98:0/2405385773 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a4005080 con 0x7f13bc0c9980 2024-03-29T10:55:24.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.662+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13c40cac70 con 0x7f13bc0c9980 2024-03-29T10:55:24.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.662+0000 7f13ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:24.643 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:24.662+0000 7f13ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:24.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.016+0000 7f13c8832640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:24.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.016+0000 7f13c8832640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:24.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.016+0000 7f13c8832640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:24.997 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.016+0000 7f13c8832640 1 -- 172.21.15.98:0/2405385773 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13a4005330 con 0x7f13bc0c9980 2024-03-29T10:55:24.999 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:55:24.999 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:54:39.048768+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:39.048768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:24.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.018+0000 7f13ab7fe640 1 -- 172.21.15.98:0/2405385773 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2920969763) 0x7f13c40ccba0 con 0x7f13bc0c9980 2024-03-29T10:55:24.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.018+0000 7f13ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:24.999 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.018+0000 7f13ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:25.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.021+0000 7f13c8832640 10 librados: watch_flush enter 2024-03-29T10:55:25.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.021+0000 7f13c8832640 10 librados: watch_flush exit 2024-03-29T10:55:25.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.021+0000 7f13c8832640 20 client.4670.objecter shutdown clearing up homeless session... 2024-03-29T10:55:25.002 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.021+0000 7f13c8832640 10 client.4670.objecter successfully canceled tick 2024-03-29T10:55:25.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.023+0000 7f13c8832640 1 -- 172.21.15.98:0/2405385773 >> v1:172.21.15.98:6816/1481799426 conn(0x7f13b006df60 legacy=0x7f13b0070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:25.003 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.023+0000 7f13c8832640 10 monclient: shutdown 2024-03-29T10:55:25.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.023+0000 7f13c8832640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:25.004 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.023+0000 7f13c8832640 1 -- 172.21.15.98:0/2405385773 >> v1:172.21.15.98:6790/0 conn(0x7f13bc0c9980 legacy=0x7f13bc0cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:25.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.025+0000 7f13c8832640 1 -- 172.21.15.98:0/2405385773 shutdown_connections 2024-03-29T10:55:25.006 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.025+0000 7f13c8832640 1 -- 172.21.15.98:0/2405385773 >> 172.21.15.98:0/2405385773 conn(0x7f13bc0058f0 msgr2=0x7f13bc0c68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:25.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.027+0000 7f13c8832640 1 -- 172.21.15.98:0/2405385773 shutdown_connections 2024-03-29T10:55:25.007 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.027+0000 7f13c8832640 1 -- 172.21.15.98:0/2405385773 wait complete. 2024-03-29T10:55:25.008 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:25.027+0000 7f13c8832640 1 librados: shutdown 2024-03-29T10:55:25.026 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:31.833 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:31.967 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.985+0000 7f2a2178f640 10 monclient: get_monmap_and_config 2024-03-29T10:55:31.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.985+0000 7f2a2178f640 10 monclient: build_initial_monmap 2024-03-29T10:55:31.968 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.985+0000 7f2a2178f640 10 monclient: monmap: 2024-03-29T10:55:31.968 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:55:31.968 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:31.968 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:55:31.987571+0000 2024-03-29T10:55:31.968 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:55:31.987571+0000 2024-03-29T10:55:31.968 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:31.968 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:31.968 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:55:31.968 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-c 2024-03-29T10:55:31.968 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-b 2024-03-29T10:55:31.968 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:31.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.986+0000 7f2a2178f640 1 Processor -- start 2024-03-29T10:55:31.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.986+0000 7f2a2178f640 1 -- start start 2024-03-29T10:55:31.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.986+0000 7f2a2178f640 10 monclient: init 2024-03-29T10:55:31.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.987+0000 7f2a2178f640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:31.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.987+0000 7f2a2178f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:31.971 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.987+0000 7f2a2178f640 10 monclient(hunting): picked mon.noname-b con 0x7f2a1c0b1f60 addr v1:172.21.15.98:6790/0 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.987+0000 7f2a2178f640 10 monclient(hunting): picked mon.noname-c con 0x7f2a1c071eb0 addr v1:172.21.15.189:6789/0 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.987+0000 7f2a2178f640 10 monclient(hunting): picked mon.noname-a con 0x7f2a1c143d10 addr v1:172.21.15.98:6789/0 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.987+0000 7f2a2178f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c06ef80 con 0x7f2a1c143d10 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.987+0000 7f2a2178f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c06f260 con 0x7f2a1c071eb0 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.987+0000 7f2a2178f640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c0b40c0 con 0x7f2a1c0b1f60 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.987+0000 7f2a2178f640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.987+0000 7f2a2178f640 10 monclient(hunting): authenticate will time out at 1552.185811s 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.988+0000 7f2a1affd640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f2a1c0b1f60 0x7f2a1c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:33092/0 (socket says 172.21.15.98:33092) 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.988+0000 7f2a1affd640 1 -- 172.21.15.98:0/1865518803 learned_addr learned my addr 172.21.15.98:0/1865518803 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.988+0000 7f2a19ffb640 1 -- 172.21.15.98:0/1865518803 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864720052 0 0) 0x7f2a1c0b40c0 con 0x7f2a1c0b1f60 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.988+0000 7f2a19ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.988+0000 7f2a19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.988+0000 7f2a19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:31.972 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.988+0000 7f2a19ffb640 10 monclient(hunting): my global_id is 4673 2024-03-29T10:55:31.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.988+0000 7f2a19ffb640 1 -- 172.21.15.98:0/1865518803 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a04003520 con 0x7f2a1c0b1f60 2024-03-29T10:55:31.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.988+0000 7f2a19ffb640 1 -- 172.21.15.98:0/1865518803 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1649966094 0 0) 0x7f2a04003520 con 0x7f2a1c0b1f60 2024-03-29T10:55:31.973 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.988+0000 7f2a19ffb640 1 -- 172.21.15.98:0/1865518803 >> v1:172.21.15.189:6789/0 conn(0x7f2a1c071eb0 legacy=0x7f2a1c146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:31.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 1 -- 172.21.15.98:0/1865518803 >> v1:172.21.15.98:6789/0 conn(0x7f2a1c143d10 legacy=0x7f2a1c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:55:31.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 1 monclient: found mon.noname-b 2024-03-29T10:55:31.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:31.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.98:6790/0 2024-03-29T10:55:31.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 1 -- 172.21.15.98:0/1865518803 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c155d70 con 0x7f2a1c0b1f60 2024-03-29T10:55:31.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 10 monclient: _finish_auth 0 2024-03-29T10:55:31.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:55:31.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:31.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 1 -- 172.21.15.98:0/1865518803 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2a14001a10 con 0x7f2a1c0b1f60 2024-03-29T10:55:31.978 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 10 monclient: dump: 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 1 -- 172.21.15.98:0/1865518803 <== mon.2 v1:172.21.15.98:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a14001be0 con 0x7f2a1c0b1f60 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:31.979 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 1 -- 172.21.15.98:0/1865518803 <== mon.2 v1:172.21.15.98:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2a14002030 con 0x7f2a1c0b1f60 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.990+0000 7f2a19ffb640 10 monclient: dump: 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:31.980 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.991+0000 7f2a2178f640 5 monclient: authenticate success, global_id 4673 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.991+0000 7f2a2178f640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.991+0000 7f2a2178f640 4 set_mon_vals no callback set 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.992+0000 7f2a2178f640 10 monclient: shutdown 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.992+0000 7f2a2178f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.992+0000 7f2a2178f640 1 -- 172.21.15.98:0/1865518803 >> v1:172.21.15.98:6790/0 conn(0x7f2a1c0b1f60 legacy=0x7f2a1c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.992+0000 7f2a2178f640 1 -- 172.21.15.98:0/1865518803 shutdown_connections 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.992+0000 7f2a2178f640 1 -- 172.21.15.98:0/1865518803 >> 172.21.15.98:0/1865518803 conn(0x7f2a1c066a00 msgr2=0x7f2a1c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.992+0000 7f2a2178f640 1 -- 172.21.15.98:0/1865518803 shutdown_connections 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.992+0000 7f2a2178f640 1 -- 172.21.15.98:0/1865518803 wait complete. 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.993+0000 7f2a2178f640 10 monclient: build_initial_monmap 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.993+0000 7f2a2178f640 10 monclient: monmap: 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-03-29T10:55:31.981 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-03-29T10:55:31.982 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:31.982 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:31.982 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a 2024-03-29T10:55:31.982 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.noname-b 2024-03-29T10:55:31.982 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.noname-c 2024-03-29T10:55:31.982 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:31.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.993+0000 7f2a2178f640 1 librados: starting msgr at 2024-03-29T10:55:31.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.993+0000 7f2a2178f640 1 librados: starting objecter 2024-03-29T10:55:31.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.993+0000 7f2a2178f640 1 Processor -- start 2024-03-29T10:55:31.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.994+0000 7f2a2178f640 1 -- start start 2024-03-29T10:55:31.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.994+0000 7f2a2178f640 1 librados: setting wanted keys 2024-03-29T10:55:31.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.994+0000 7f2a2178f640 1 librados: calling monclient init 2024-03-29T10:55:31.982 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.994+0000 7f2a2178f640 10 monclient: init 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.994+0000 7f2a2178f640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.994+0000 7f2a2178f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.995+0000 7f2a2178f640 10 monclient(hunting): picked mon.noname-a con 0x7f2a1c0b1f60 addr v1:172.21.15.98:6789/0 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.995+0000 7f2a2178f640 10 monclient(hunting): picked mon.noname-c con 0x7f2a1c143d10 addr v1:172.21.15.98:6790/0 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.995+0000 7f2a1a7fc640 1 --1- >> v1:172.21.15.98:6790/0 conn(0x7f2a1c143d10 0x7f2a1c1deed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6790/0 says I am v1:172.21.15.98:33572/0 (socket says 172.21.15.98:33572) 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a2178f640 10 monclient(hunting): picked mon.noname-b con 0x7f2a1c1548a0 addr v1:172.21.15.189:6789/0 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a2178f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c070c10 con 0x7f2a1c0b1f60 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a2178f640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c1394e0 con 0x7f2a1c1548a0 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a2178f640 1 -- --> v1:172.21.15.98:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c1397c0 con 0x7f2a1c143d10 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a2178f640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a2178f640 10 monclient(hunting): authenticate will time out at 1552.194131s 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a1a7fc640 1 -- 172.21.15.98:0/4106999060 learned_addr learned my addr 172.21.15.98:0/4106999060 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 10 client.?.objecter ms_handle_connect 0x7f2a1c143d10 2024-03-29T10:55:31.983 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:31.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 <== mon.2 v1:172.21.15.98:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411249986 0 0) 0x7f2a1c1397c0 con 0x7f2a1c143d10 2024-03-29T10:55:31.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:31.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:31.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:31.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 10 monclient(hunting): my global_id is 4676 2024-03-29T10:55:31.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 --> v1:172.21.15.98:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a08004eb0 con 0x7f2a1c143d10 2024-03-29T10:55:31.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 10 client.?.objecter ms_handle_connect 0x7f2a1c0b1f60 2024-03-29T10:55:31.984 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:31.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 <== mon.2 v1:172.21.15.98:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 635316140 0 0) 0x7f2a08004eb0 con 0x7f2a1c143d10 2024-03-29T10:55:31.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 --> v1:172.21.15.98:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a1c1397c0 con 0x7f2a1c143d10 2024-03-29T10:55:31.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 <== mon.2 v1:172.21.15.98:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2a10001870 con 0x7f2a1c143d10 2024-03-29T10:55:31.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:31.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:31.986 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 10 monclient(hunting): dump: 2024-03-29T10:55:31.986 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.996+0000 7f2a037fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905958005 0 0) 0x7f2a1c070c10 con 0x7f2a1c0b1f60 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 10 monclient(hunting): _negotiate 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:31.987 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 10 monclient(hunting): my global_id is 4605 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a08004eb0 con 0x7f2a1c0b1f60 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 <== mon.2 v1:172.21.15.98:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4013136375 0 0) 0x7f2a1c1397c0 con 0x7f2a1c143d10 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 >> v1:172.21.15.189:6789/0 conn(0x7f2a1c1548a0 legacy=0x7f2a1c1df5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 >> v1:172.21.15.98:6789/0 conn(0x7f2a1c0b1f60 legacy=0x7f2a1c13a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 10 monclient: _finish_hunting 0 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 1 monclient: found mon.c 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 --> v1:172.21.15.98:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c139aa0 con 0x7f2a1c143d10 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 10 monclient: _finish_auth 0 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 10 monclient: _check_auth_tickets 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a2178f640 5 monclient: authenticate success, global_id 4676 2024-03-29T10:55:31.988 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a2178f640 10 monclient: _renew_subs 2024-03-29T10:55:31.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a2178f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:31.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a2178f640 1 -- 172.21.15.98:0/4106999060 --> v1:172.21.15.98:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a1c139d80 con 0x7f2a1c143d10 2024-03-29T10:55:31.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a2178f640 10 client.4676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:31.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a2178f640 10 monclient: _renew_subs 2024-03-29T10:55:31.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a2178f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:31.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a2178f640 1 -- 172.21.15.98:0/4106999060 --> v1:172.21.15.98:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1c1554f0 con 0x7f2a1c143d10 2024-03-29T10:55:31.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.997+0000 7f2a2178f640 1 librados: init done 2024-03-29T10:55:31.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.998+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 <== mon.2 v1:172.21.15.98:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a10004900 con 0x7f2a1c143d10 2024-03-29T10:55:31.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.998+0000 7f2a037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:31.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.998+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 <== mon.2 v1:172.21.15.98:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1664827347 0 0) 0x7f2a10004c10 con 0x7f2a1c143d10 2024-03-29T10:55:31.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.998+0000 7f2a037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:31.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.998+0000 7f2a037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:31.989 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.998+0000 7f2a037fe640 10 monclient: dump: 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:fsid 556f8916-2d97-4161-a370-2566dd79bd5f 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:created 2024-03-29T10:43:55.920917+0000 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.a 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.189:6789/0 mon.b 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:2: v1:172.21.15.98:6790/0 mon.c 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr: 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.998+0000 7f2a197fa640 4 set_mon_vals no callback set 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.999+0000 7f2a2178f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.999+0000 7f2a2178f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.999+0000 7f2a2178f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:31.990 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:31.999+0000 7f2a2178f640 1 -- 172.21.15.98:0/4106999060 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29fc005080 con 0x7f2a1c143d10 2024-03-29T10:55:31.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.000+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 <== mon.2 v1:172.21.15.98:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1552141041 0 0) 0x7f2a1001abe0 con 0x7f2a1c143d10 2024-03-29T10:55:31.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.000+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 <== mon.2 v1:172.21.15.98:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2292924465 0 0) 0x7f2a10084b30 con 0x7f2a1c143d10 2024-03-29T10:55:31.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.000+0000 7f2a037fe640 10 client.4676.objecter ms_dispatch 0x7f2a1c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:55:31.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.000+0000 7f2a037fe640 3 client.4676.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:55:31.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.000+0000 7f2a037fe640 3 client.4676.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:55:31.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.000+0000 7f2a037fe640 20 client.4676.objecter dump_active .. 0 homeless 2024-03-29T10:55:31.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.000+0000 7f2a1affd640 10 monclient: get_auth_request con 0x7f2a0806dad0 auth_method 0 2024-03-29T10:55:31.991 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.001+0000 7f2a037fe640 10 client.4676.objecter ms_handle_connect 0x7f2a0806dad0 2024-03-29T10:55:31.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.008+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 <== mon.2 v1:172.21.15.98:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a10050240 con 0x7f2a1c143d10 2024-03-29T10:55:31.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.008+0000 7f2a037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:31.992 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.008+0000 7f2a037fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:32.297 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.315+0000 7f2a2178f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:32.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.315+0000 7f2a2178f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:32.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.316+0000 7f2a2178f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.98:6790/0 2024-03-29T10:55:32.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.316+0000 7f2a2178f640 1 -- 172.21.15.98:0/4106999060 --> v1:172.21.15.98:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29fc005330 con 0x7f2a1c143d10 2024-03-29T10:55:32.298 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.317+0000 7f2a037fe640 1 -- 172.21.15.98:0/4106999060 <== mon.2 v1:172.21.15.98:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2920969763) 0x7f2a10052340 con 0x7f2a1c143d10 2024-03-29T10:55:32.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.317+0000 7f2a037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:32.299 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.317+0000 7f2a037fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:32.299 INFO:teuthology.misc.health.smithi098.stdout: 2024-03-29T10:55:32.299 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"556f8916-2d97-4161-a370-2566dd79bd5f","created":"2024-03-29T10:44:34.115037+0000","modified":"2024-03-29T10:54:39.048768+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:39.048768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"04bb1a53-61cb-479f-add3-3f9964d54955","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bd57627e-f989-44a1-ba96-4fb20f6191ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"89e48e12-41c1-408c-b994-f567d9c3c139","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"197cfc95-2d74-4ca4-9c1c-97e1fdfd5d30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d6b02fb-0c49-4df6-9d91-c1600d8d6609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1083e880-407b-4f66-9207-c62e48fd52f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cfeca019-a949-4d0d-aafc-afd40c7f39fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2132347f-1364-46a0-9722-bf70172452ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:32.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.319+0000 7f2a2178f640 10 librados: watch_flush enter 2024-03-29T10:55:32.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.319+0000 7f2a2178f640 10 librados: watch_flush exit 2024-03-29T10:55:32.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.319+0000 7f2a2178f640 20 client.4676.objecter shutdown clearing up homeless session... 2024-03-29T10:55:32.302 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.319+0000 7f2a2178f640 10 client.4676.objecter successfully canceled tick 2024-03-29T10:55:32.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.320+0000 7f2a2178f640 1 -- 172.21.15.98:0/4106999060 >> v1:172.21.15.98:6816/1481799426 conn(0x7f2a0806dad0 legacy=0x7f2a0806ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:32.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.320+0000 7f2a2178f640 10 monclient: shutdown 2024-03-29T10:55:32.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.320+0000 7f2a2178f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:32.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.320+0000 7f2a2178f640 1 -- 172.21.15.98:0/4106999060 >> v1:172.21.15.98:6790/0 conn(0x7f2a1c143d10 legacy=0x7f2a1c1deed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:32.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.320+0000 7f2a2178f640 1 -- 172.21.15.98:0/4106999060 shutdown_connections 2024-03-29T10:55:32.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.320+0000 7f2a2178f640 1 -- 172.21.15.98:0/4106999060 >> 172.21.15.98:0/4106999060 conn(0x7f2a1c066a00 msgr2=0x7f2a1c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:32.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.320+0000 7f2a2178f640 1 -- 172.21.15.98:0/4106999060 shutdown_connections 2024-03-29T10:55:32.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.320+0000 7f2a2178f640 1 -- 172.21.15.98:0/4106999060 wait complete. 2024-03-29T10:55:32.303 INFO:teuthology.misc.health.smithi098.stderr:2024-03-29T10:55:32.320+0000 7f2a2178f640 1 librados: shutdown 2024-03-29T10:55:32.316 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:32.316 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:55:32.318 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T10:55:32.319 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T10:55:32.319 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:55:32.352 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T10:55:32.352 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T10:55:32.353 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:55:32.422 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T10:55:32.423 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T10:55:32.423 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:55:32.492 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T10:55:32.493 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T10:55:32.493 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:55:32.564 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T10:55:32.564 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T10:55:32.564 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:55:32.606 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T10:55:32.607 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T10:55:32.607 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:55:32.637 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T10:55:32.638 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T10:55:32.638 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:55:32.715 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T10:55:32.715 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T10:55:32.715 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:55:32.792 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T10:55:32.792 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T10:55:32.792 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T10:55:32.792 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:55:32.817 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T10:55:32.817 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T10:55:32.817 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:55:32.856 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T10:55:32.856 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T10:55:32.856 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T10:55:32.856 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:55:32.869 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T10:55:32.869 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T10:55:32.869 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:55:32.921 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T10:55:32.921 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T10:55:32.921 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:55:32.932 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T10:55:32.932 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T10:55:32.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T10:55:33.057 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T10:55:33.057 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T10:55:36.671 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T10:55:36.671 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T10:55:36.764 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T10:55:36.765 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T10:55:36.813 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T10:55:36.813 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T10:55:36.864 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi189.front.sepia.ceph.com 2024-03-29T10:55:36.864 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T10:55:39.405 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi189.front.sepia.ceph.com 2024-03-29T10:55:39.405 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T10:55:39.482 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi189.front.sepia.ceph.com 2024-03-29T10:55:39.482 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T10:55:39.557 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi189.front.sepia.ceph.com 2024-03-29T10:55:39.558 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T10:55:39.617 INFO:tasks.ceph:Archiving mon data... 2024-03-29T10:55:39.619 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649/data/mon.a.tgz 2024-03-29T10:55:39.619 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-29T10:55:39.638 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.QJvB8DmDni 2024-03-29T10:55:39.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.QJvB8DmDni 2024-03-29T10:55:39.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.QJvB8DmDni 2024-03-29T10:55:39.861 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.QJvB8DmDni is 126KB 2024-03-29T10:55:39.915 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.QJvB8DmDni 2024-03-29T10:55:39.930 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649/data/mon.c.tgz 2024-03-29T10:55:39.931 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-29T10:55:39.986 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.DUCPZPBTie 2024-03-29T10:55:39.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.DUCPZPBTie 2024-03-29T10:55:40.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.DUCPZPBTie 2024-03-29T10:55:40.162 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.DUCPZPBTie is 126KB 2024-03-29T10:55:40.213 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.DUCPZPBTie 2024-03-29T10:55:40.228 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649/data/mon.b.tgz 2024-03-29T10:55:40.228 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-03-29T10:55:40.242 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.9uNHoqaS25 2024-03-29T10:55:40.242 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.9uNHoqaS25 2024-03-29T10:55:40.340 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.9uNHoqaS25 2024-03-29T10:55:40.424 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.9uNHoqaS25 is 126KB 2024-03-29T10:55:40.476 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.9uNHoqaS25 2024-03-29T10:55:40.490 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T10:55:40.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:55:40.493 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:55:40.572 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:55:40.612 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:55:40.612 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T10:55:40.614 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649/remote/smithi098/crash 2024-03-29T10:55:40.615 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:55:40.641 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649/remote/smithi189/crash 2024-03-29T10:55:40.642 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:55:40.679 INFO:tasks.ceph:Compressing logs... 2024-03-29T10:55:40.679 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:55:40.686 DEBUG:teuthology.orchestra.run.smithi189:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:55:40.754 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39890.log 2024-03-29T10:55:40.754 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39773.log 2024-03-29T10:55:40.754 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39890.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T10:55:40.754 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41342.log 2024-03-29T10:55:40.754 INFO:teuthology.orchestra.run.smithi098.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39890.log.gz 2024-03-29T10:55:40.754 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.a.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39773.log.gz 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41732.log 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39071.log 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39929.log 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41342.log.gz 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41732.log.gz 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40913.log 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39305.log 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39929.log: gzip 86.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.39071.log.gz --verbose 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/ceph-client.admin.40248.log 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40913.log: /var/log/ceph/ceph-client.admin.39305.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39929.log.gz 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38837.log 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40913.log.gz 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41225.log 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40248.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39305.log.gz 2024-03-29T10:55:40.755 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38993.log 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41225.log: 86.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.40248.log.gz -5 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41537.log 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38837.log.gz 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38993.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28798.log 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41225.log.gz 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41576.log 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38993.log.gz 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40796.log 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41537.log.gz 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.tmp-client.admin.28798.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28798.log.gz 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41654.log 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40170.log 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41576.log.gz 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38523.log 2024-03-29T10:55:40.756 INFO:teuthology.orchestra.run.smithi098.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40796.log.gz 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41654.log.gz 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40991.log 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40170.log: /var/log/ceph/ceph-client.admin.38523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38759.log 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40874.log 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40170.log.gz 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38523.log.gz 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40085.log 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40991.log.gz 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38445.log 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40874.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38759.log.gz 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38367.log 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40874.log.gz 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38445.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:55:40.757 INFO:teuthology.orchestra.run.smithi098.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40085.log.gz 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38367.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38445.log.gz 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40404.log 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.audit.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38367.log.gz 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38915.log 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40952.log 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38915.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41186.log 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.40404.log.gz 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38915.log.gz 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41147.log 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40952.log: gzip -5/var/log/ceph/ceph-client.admin.41186.log: --verbose -- /var/log/ceph/ceph-client.admin.41420.log 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39422.log 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40952.log.gz 2024-03-29T10:55:40.758 INFO:teuthology.orchestra.run.smithi098.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41186.log.gz 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41030.log 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41147.log.gz 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39422.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41420.log.gz 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39851.log 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39422.log.gz 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40482.log 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41030.log.gz 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41069.log 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39851.log.gz 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38289.log 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40131.log 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41069.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40482.log.gz 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39578.log 2024-03-29T10:55:40.759 INFO:teuthology.orchestra.run.smithi098.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41069.log.gz 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41264.log 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38289.log.gz 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39578.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40131.log.gz 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39617.log 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40326.log 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41264.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39578.log.gz 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41381.log 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40326.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41264.log.gz 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40716.log 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39617.log.gz 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41693.log 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41381.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40326.log.gz 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39968.log 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41381.log.gz 2024-03-29T10:55:40.760 INFO:teuthology.orchestra.run.smithi098.stderr: 86.8%/var/log/ceph/ceph-client.admin.41693.log: -- replaced with /var/log/ceph/ceph-client.admin.40716.log.gz 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39227.log 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr: 86.6%/var/log/ceph/ceph-osd.3.log: -- replaced with /var/log/ceph/ceph-client.admin.41693.log.gz 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39344.log 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39968.log.gz 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40365.log 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35000.log 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39227.log.gz 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38681.log 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39344.log.gz 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38642.log 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40365.log.gz 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40287.log 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35000.log.gz 2024-03-29T10:55:40.761 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38642.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38681.log.gz 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41108.log 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39500.log 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38642.log.gz 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40638.log 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40287.log.gz 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39500.log: gzip -5 --verbose -- 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41108.log.gz 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr: /var/log/ceph/ceph-client.admin.40443.log 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40638.log: gzip -5 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39500.log.gz 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40443.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40638.log.gz 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39110.log 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40443.log.gz 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40007.log 2024-03-29T10:55:40.762 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40677.log 2024-03-29T10:55:40.763 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40007.log: gzip -5 --verbose 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39110.log.gz 2024-03-29T10:55:40.763 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/ceph-client.admin.40599.log 2024-03-29T10:55:40.763 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40677.log: gzip 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40007.log.gz 2024-03-29T10:55:40.763 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39695.log 2024-03-29T10:55:40.763 INFO:teuthology.orchestra.run.smithi098.stderr: 86.1%/var/log/ceph/ceph-client.admin.40599.log: -- replaced with /var/log/ceph/ceph-client.admin.40677.log.gz 2024-03-29T10:55:40.763 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39383.log 2024-03-29T10:55:40.763 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39695.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40599.log.gz 2024-03-29T10:55:40.763 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41498.log 2024-03-29T10:55:40.763 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39383.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39695.log.gz 2024-03-29T10:55:40.763 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39656.log 2024-03-29T10:55:40.763 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41498.log: 86.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40757.log 2024-03-29T10:55:40.763 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39383.log.gz 2024-03-29T10:55:40.763 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34897.log 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41498.log.gz 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5%/var/log/ceph/ceph-client.admin.40757.log: -- replaced with /var/log/ceph/ceph-client.admin.39656.log.gz 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40835.log 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39266.log 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34897.log.gz 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40757.log.gz 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40521.log 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40835.log.gz 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38876.log 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39266.log: /var/log/ceph/ceph-client.admin.40521.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39266.log.gz 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40521.log.gz 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41615.log 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41303.log 2024-03-29T10:55:40.764 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41615.log: 85.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40209.log 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38876.log.gz 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41615.log.gz 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39188.log 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40209.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr: 85.8%/var/log/ceph/ceph-client.admin.39188.log: -- replaced with /var/log/ceph/ceph-client.admin.41303.log.gz 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40209.log.gz 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38720.log 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39188.log.gz 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.38720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39461.log 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.2.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38720.log.gz 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39149.log 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39461.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr: 86.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38484.log 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39461.log.gz 2024-03-29T10:55:40.765 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40046.log 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39149.log: /var/log/ceph/ceph-client.admin.38484.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38484.log.gz 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39149.log.gz 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39812.log 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38406.log 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39812.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40046.log.gz 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38798.log 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38406.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39812.log.gz 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:gzip 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38406.log.gz 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39032.log 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.38798.log: /var/log/ceph/ceph-client.admin.40560.log 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39032.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38798.log.gz 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39539.log 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39032.log.gz 2024-03-29T10:55:40.766 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.40560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34606.log 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40560.log.gz/var/log/ceph/ceph-client.admin.39539.log: 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38954.log 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34606.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39539.log.gz 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38603.log 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.34606.log.gz 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41459.log 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38603.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38954.log.gz 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38603.log.gz 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38564.log 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38328.log 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.41459.log: /var/log/ceph/ceph-client.admin.38564.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41459.log.gz 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38564.log.gz 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39734.log 2024-03-29T10:55:40.767 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38328.log: /var/log/ceph/ceph-mon.c.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38328.log.gz 2024-03-29T10:55:40.768 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.39734.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39734.log.gz 2024-03-29T10:55:40.771 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T10:55:40.771 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34652.log 2024-03-29T10:55:40.771 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34612.log 2024-03-29T10:55:40.771 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph-client.admin.34652.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34652.log.gz 2024-03-29T10:55:40.771 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34812.log 2024-03-29T10:55:40.771 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph-client.admin.34612.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:55:40.771 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph-client.admin.34812.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T10:55:40.771 INFO:teuthology.orchestra.run.smithi189.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34612.log.gz 2024-03-29T10:55:40.771 INFO:teuthology.orchestra.run.smithi189.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34812.log.gz 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34892.log 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34852.log 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph-client.admin.34892.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph-client.admin.34852.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34892.log.gzgzip 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34852.log.gz 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34772.log 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34732.log 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph-client.admin.34772.log: /var/log/ceph/ceph-client.admin.34732.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34772.log.gz 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34732.log.gz 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:55:40.772 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T10:55:40.773 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34692.log 2024-03-29T10:55:40.773 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph-mon.b.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:55:40.773 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/ceph-client.admin.34692.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34692.log.gz 2024-03-29T10:55:40.773 INFO:teuthology.orchestra.run.smithi189.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T10:55:40.781 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T10:55:52.856 INFO:teuthology.orchestra.run.smithi098.stderr: 94.6% 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T10:55:52.856 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:55:53.141 INFO:teuthology.orchestra.run.smithi189.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T10:55:53.330 INFO:teuthology.orchestra.run.smithi189.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T10:55:57.125 INFO:teuthology.orchestra.run.smithi098.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T10:55:57.158 INFO:teuthology.orchestra.run.smithi098.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T10:55:57.587 INFO:teuthology.orchestra.run.smithi189.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T10:55:57.616 INFO:teuthology.orchestra.run.smithi189.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T10:56:33.841 INFO:teuthology.orchestra.run.smithi098.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T10:56:37.474 INFO:teuthology.orchestra.run.smithi098.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T10:56:37.476 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-29T10:56:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m56.779s 2024-03-29T10:56:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:user 2m41.036s 2024-03-29T10:56:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m5.722s 2024-03-29T10:56:48.520 INFO:teuthology.orchestra.run.smithi189.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T10:56:48.521 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-29T10:56:48.521 INFO:teuthology.orchestra.run.smithi189.stderr:real 1m7.786s 2024-03-29T10:56:48.522 INFO:teuthology.orchestra.run.smithi189.stderr:user 2m1.770s 2024-03-29T10:56:48.522 INFO:teuthology.orchestra.run.smithi189.stderr:sys 0m4.469s 2024-03-29T10:56:48.522 INFO:tasks.ceph:Archiving logs... 2024-03-29T10:56:48.523 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649/remote/smithi098/log 2024-03-29T10:56:48.524 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:57:11.966 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649/remote/smithi189/log 2024-03-29T10:57:11.967 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:57:31.358 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:57:31.559 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c306800ae5d54566a4a7cd9a21c60896 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:57:31.562 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T10:57:31.574 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T10:57:31.588 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T10:57:31.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:57:31.602 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:57:31.676 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T10:57:31.684 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T10:57:31.685 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:57:31.687 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:57:31.700 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:57:31.701 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:57:31.706 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:57:31.706 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-03-29T10:57:31.706 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 +435us[ +435us] +/- 75ms 2024-03-29T10:57:31.706 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -191us[ -200us] +/- 55ms 2024-03-29T10:57:31.706 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 -186us[ -186us] +/- 86ms 2024-03-29T10:57:31.706 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:57:31.707 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:57:31.707 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-29T10:57:31.707 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +424us[ +424us] +/- 75ms 2024-03-29T10:57:31.707 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -178us[ -178us] +/- 55ms 2024-03-29T10:57:31.707 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 -199us[ -199us] +/- 86ms 2024-03-29T10:57:31.707 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:57:31.707 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T10:57:31.716 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T10:57:31.716 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T10:57:31.725 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:57:31.815 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:57:31.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:57:31.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:57:31.910 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:57:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:57:31.940 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T10:57:31.949 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T10:57:31.961 INFO:teuthology.task.internal:Duration was 1335.798959 seconds 2024-03-29T10:57:31.961 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T10:57:31.975 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T10:57:31.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:57:31.978 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:57:32.020 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:57:32.023 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:57:32.266 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T10:57:32.266 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-03-29T10:57:32.267 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:57:32.326 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-03-29T10:57:32.326 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:57:32.347 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T10:57:32.348 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:57:32.368 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:57:32.416 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T10:57:32.416 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:57:32.436 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:57:32.553 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T10:57:32.562 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T10:57:32.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:57:32.589 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:57:32.616 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T10:57:32.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:57:32.631 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:57:32.655 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-03-29T10:57:32.682 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-03-29T10:57:32.703 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:57:32.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:57:32.729 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:57:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:57:32.757 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T10:57:32.767 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T10:57:32.768 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649/remote/smithi098 2024-03-29T10:57:32.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:57:32.805 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649/remote/smithi189 2024-03-29T10:57:32.805 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:57:32.840 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T10:57:32.841 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:57:32.843 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:57:32.890 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T10:57:32.901 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T10:57:32.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T10:57:32.913 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T10:57:32.914 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:57:32.916 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:57:32.928 INFO:teuthology.orchestra.run.smithi098.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:57 /home/ubuntu/cephtest 2024-03-29T10:57:32.945 INFO:teuthology.orchestra.run.smithi189.stdout: 265625 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:57 /home/ubuntu/cephtest 2024-03-29T10:57:32.946 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T10:57:32.958 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T10:57:32.985 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T10:57:33.012 DEBUG:teuthology.nuke:shortname: smithi098 2024-03-29T10:57:33.013 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:57:33.036 DEBUG:teuthology.nuke:shortname: smithi189 2024-03-29T10:57:33.037 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:57:33.040 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:26:54.716326', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOs/ndKlvjGdBzG5f1iS4hR4q57LTrq2aAzoPkLTTFv0bqaek2BKZukHVQyRNNKa+it/lpQSm/z9NtWS+GyXZ3U='} 2024-03-29T10:57:33.065 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622649', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:26:54.717795', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwqt2fg/uG9aD+S9vmfcYKVAE4utlnhEcAyqIpN44VF5msL/7BqUAgmNf3en7xQFd8nKxjiBt19L8OdGwTYbNM='} 2024-03-29T10:57:33.085 INFO:teuthology.orchestra.console.smithi098:Power off 2024-03-29T10:57:33.086 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:57:33.117 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-03-29T10:57:33.117 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:57:33.131 INFO:teuthology.orchestra.console.smithi189:Power off 2024-03-29T10:57:33.132 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:57:33.141 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-29T10:57:33.156 DEBUG:teuthology.orchestra.console.smithi189:power off output: Chassis Power Control: Down/Off 2024-03-29T10:57:33.156 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:57:33.179 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-29T10:57:37.142 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:57:37.180 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:57:37.253 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-29T10:57:37.292 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-29T10:57:41.257 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:57:41.293 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:57:41.368 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-29T10:57:41.404 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-29T10:57:41.469 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-03-29T10:57:45.406 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:57:45.516 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is off 2024-03-29T10:57:45.617 INFO:teuthology.orchestra.console.smithi189:Power off completed 2024-03-29T10:57:45.722 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1335.798959493637 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c306800ae5d54566a4a7cd9a21c60896 status: fail success: false 2024-03-29T10:57:45.722 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:57:45.799 INFO:teuthology.run:FAIL